2020-11-16T20:17:06.500 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-16T20:17:06.501 DEBUG:teuthology.run:Teuthology command: teuthology --block --verbose --owner scheduled_kchai@teuthology --name kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi --description rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} --lock --archive /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154 -- /tmp/teuthology-worker.6as4y9y6.tmp 2020-11-16T20:17:06.528 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T20:17:06.599 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154 branch: wip-kefu-testing-2020-11-16-1845 description: rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} email: tchaikov@gmail.com first_in_suite: false job_id: '5631154' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-kefu-testing-2020-11-16-1845 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - Health detail - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - Health detail mon_bind_msgr2: false sha1: 911cc35794d871071a8bebfb0919482263b1d18b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 911cc35794d871071a8bebfb0919482263b1d18b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2020-11-16-1845 sha1: 911cc35794d871071a8bebfb0919482263b1d18b owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 sha1: 911cc35794d871071a8bebfb0919482263b1d18b sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2020-11-16-1845 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-16-1845/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 911cc35794d871071a8bebfb0919482263b1d18b tasks: - install: null - ceph: conf: osd: osd min pg log entries: 5 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - osd_backfill: null teuthology_branch: master timestamp: 2020-11-16_12:11:03 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.11580 2020-11-16T20:17:06.599 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-16-1845/qa; will attempt to use it 2020-11-16T20:17:06.600 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-16-1845/qa/tasks 2020-11-16T20:17:06.601 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-16T20:17:06.603 INFO:teuthology.task.internal:Checking packages... 2020-11-16T20:17:06.630 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '911cc35794d871071a8bebfb0919482263b1d18b' 2020-11-16T20:17:06.631 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-16T20:17:06.632 INFO:teuthology.packaging:ref: None 2020-11-16T20:17:06.632 INFO:teuthology.packaging:tag: None 2020-11-16T20:17:06.632 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-11-16-1845 2020-11-16T20:17:06.633 INFO:teuthology.packaging:sha1: 911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T20:17:06.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2020-11-16-1845 2020-11-16T20:17:06.804 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7219.g911cc357 2020-11-16T20:17:06.805 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-16T20:17:06.815 INFO:teuthology.task.internal:no buildpackages task found 2020-11-16T20:17:06.816 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-16T20:17:06.830 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-16T20:17:06.830 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T20:17:06.882 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 1, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154'} 2020-11-16T20:17:06.907 DEBUG:teuthology.lock.ops:locked ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T20:17:06.934 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0fc7219950>, signals=[15]) 2020-11-16T20:17:07.124 INFO:teuthology.provision.fog.smithi120:Scheduling deploy of centos 8.1 2020-11-16T20:17:07.446 INFO:teuthology.orchestra.console:Power off smithi120 2020-11-16T20:17:07.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-16T20:17:07.475 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T20:17:11.499 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T20:17:15.613 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T20:17:19.727 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T20:17:19.938 INFO:teuthology.orchestra.console:Power off for smithi120 completed 2020-11-16T20:17:20.039 INFO:teuthology.orchestra.console:Power on smithi120 2020-11-16T20:17:20.040 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-16T20:17:20.060 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T20:17:24.084 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T20:17:28.332 INFO:teuthology.orchestra.console:Power on for smithi120 completed 2020-11-16T20:17:28.433 INFO:teuthology.provision.fog.smithi120:Waiting for deploy to finish 2020-11-16T20:20:03.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:20:27.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:20:37.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:20:46.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:20:55.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:22:01.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:22:05.626 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-16T20:22:05.628 INFO:teuthology.orchestra.run.smithi120:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-16T20:22:05.917 INFO:teuthology.provision.fog.smithi120:Node is ready 2020-11-16T20:22:05.917 INFO:teuthology.orchestra.run.smithi120:> hostname 2020-11-16T20:22:05.932 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120.front.sepia.ceph.com 2020-11-16T20:22:05.932 INFO:teuthology.orchestra.run.smithi120:> grep smithi120.front.sepia.ceph.com /etc/hosts 2020-11-16T20:22:05.987 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.15.120 smithi120.front.sepia.ceph.com smithi120 2020-11-16T20:22:05.987 INFO:teuthology.orchestra.run.smithi120:> sudo hostname smithi120 2020-11-16T20:22:06.097 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i -e 's/smithi120.front.sepia.ceph.com/smithi120/g' /etc/hosts 2020-11-16T20:22:06.124 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i -e 's/172.21.15.120/172.21.15.120/g' /etc/hosts 2020-11-16T20:22:06.191 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i -e 's/smithi120.front.sepia.ceph.com/smithi120/g' /etc/hostname 2020-11-16T20:22:06.257 INFO:teuthology.orchestra.run.smithi120:> sudo hostname smithi120 2020-11-16T20:22:06.323 INFO:teuthology.provision.fog.smithi120:Deploy complete! 2020-11-16T20:22:06.323 DEBUG:teuthology.parallel:result is None 2020-11-16T20:22:10.645 INFO:teuthology.lock.ops:Checking smithi120.front.sepia.ceph.com 2020-11-16T20:22:10.646 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-16T20:22:10.663 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: set os type and version on server 2020-11-16T20:22:10.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T20:22:10.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:22:10.734 INFO:teuthology.orchestra.run.smithi120:> true 2020-11-16T20:22:10.896 INFO:teuthology.orchestra.run.smithi120:> uname -m 2020-11-16T20:22:10.950 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2020-11-16T20:22:10.951 INFO:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2020-11-16T20:22:11.005 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Linux" 2020-11-16T20:22:11.005 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8 (Core)" 2020-11-16T20:22:11.005 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2020-11-16T20:22:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2020-11-16T20:22:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8" 2020-11-16T20:22:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8" 2020-11-16T20:22:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-16T20:22:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2020-11-16T20:22:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-16T20:22:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.centos.org/" 2020-11-16T20:22:11.007 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-16T20:22:11.007 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T20:22:11.007 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-16T20:22:11.007 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-16T20:22:11.007 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-16T20:22:11.007 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-16T20:22:11.007 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T20:22:11.009 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2020-11-16T20:22:11.024 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-11-16T20:22:11.043 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgazuGy1FDOixf3HJOVsflIrbMzrTtxieWw4kKWyrMzsKquLnoPrQYrZBPCNWnRVBhde1lsPzVjsUZosfJkvpsV2FX9JZSP03t2LX05FEoXUoIev8npix8/GyOrHNszuT9k+4xPcOsgDbtuvUZ1LMExV783IKeXxYx2+w3YUkehGLxKPzDyN+Cf2DOYkTL7JPnUQASEOmZdOql5sF14IQcOhYD1CEPxeN6OfI6ndrjs6bmCcLveFYsfRJluA3FPMVa3s/ogL6+dFSFdA7vGHUpDjwCLv1P5obSJNfOiDXbpgaDyjs9juKXgAg2tlTHemn2glyCIyRFmrbvViuza9QiRZzOirDbIIAhyhGzC49g9ePxF+MrRRkJ58hy8WUuVOKBn9DLew2D3Y6yXnhfwHriNMz+XcSBV32xQ11V/aBVcQj+aZGaaKmkvAzBjrWiolpTw+knOvuP3y42vZYz4pfwcMjmAZHLmMvuNoIzE04dj/VwKIxJAvsCfq/HiGUWLCU= 2020-11-16T20:22:11.044 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T20:22:11.093 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-16T20:22:11.102 INFO:teuthology.task.internal:Saving configuration 2020-11-16T20:22:11.116 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-16T21:12:05.388 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-16T21:12:05.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-16 20:17:06.890433', 'description': '/home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBzUY13xx/B1uMMXkJBjjoivZWRbaMT6vhXZ1GLOZZGEFEmT3SoKFJnAOweNCrN4br5+ouPG4lAF9a6XCZmxEoA=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi120.front.sepia.ceph.com'} 2020-11-16T21:12:05.516 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-16T21:12:05.526 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2020-11-16T21:12:05.526 INFO:teuthology.run_tasks:Running task console_log... 2020-11-16T21:12:05.864 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0fc72137b8>, signals=[15]) 2020-11-16T21:12:05.865 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-16T21:12:05.874 INFO:teuthology.task.internal:Opening connections... 2020-11-16T21:12:05.875 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:12:05.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T21:12:05.957 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-16T21:12:05.964 INFO:teuthology.orchestra.run.smithi120:> uname -m 2020-11-16T21:12:06.121 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2020-11-16T21:12:06.121 INFO:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2020-11-16T21:12:06.175 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Linux" 2020-11-16T21:12:06.175 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8 (Core)" 2020-11-16T21:12:06.175 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2020-11-16T21:12:06.175 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2020-11-16T21:12:06.175 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8" 2020-11-16T21:12:06.176 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8" 2020-11-16T21:12:06.176 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-16T21:12:06.176 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2020-11-16T21:12:06.176 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-16T21:12:06.176 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.centos.org/" 2020-11-16T21:12:06.176 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-16T21:12:06.176 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:12:06.177 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-16T21:12:06.177 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-16T21:12:06.177 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-16T21:12:06.177 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-16T21:12:06.177 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:12:06.177 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2020-11-16T21:12:06.193 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-16T21:12:06.268 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-16T21:12:06.276 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-16T21:12:06.276 INFO:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2020-11-16T21:12:06.290 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-16T21:12:06.297 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-16T21:12:06.297 INFO:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2020-11-16T21:12:06.360 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-16T21:12:06.384 INFO:teuthology.run_tasks:Running task kernel... 2020-11-16T21:12:06.399 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}} 2020-11-16T21:12:06.399 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-16T21:12:06.400 INFO:teuthology.orchestra.run.smithi120:> uname -r 2020-11-16T21:12:06.413 INFO:teuthology.orchestra.run.smithi120.stdout:4.18.0-193.28.1.el8_2.x86_64 2020-11-16T21:12:06.414 INFO:teuthology.task.kernel:Running kernel on smithi120: 4.18.0-193.28.1.el8_2.x86_64 2020-11-16T21:12:06.414 INFO:teuthology.orchestra.run.smithi120:> sudo yum install -y kernel 2020-11-16T21:12:09.006 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - AppStream 41 MB/s | 7.0 MB 00:00 2020-11-16T21:12:10.198 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Base 22 MB/s | 2.2 MB 00:00 2020-11-16T21:12:10.629 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Extras 82 kB/s | 5.5 kB 00:00 2020-11-16T21:12:11.048 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 22 MB/s | 8.3 MB 00:00 2020-11-16T21:12:12.765 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 4.5 MB/s | 23 kB 00:00 2020-11-16T21:12:13.762 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-11-16T21:12:13.762 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2020-11-16T21:12:13.944 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:12:13.946 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:12:13.946 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:12:14.084 INFO:teuthology.orchestra.run.smithi120:> echo no | sudo yum reinstall kernel || true 2020-11-16T21:12:14.486 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:02 ago on Mon 16 Nov 2020 09:12:12 PM UTC. 2020-11-16T21:12:14.737 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-16T21:12:14.738 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-16T21:12:14.805 INFO:teuthology.orchestra.run.smithi120.stderr:Error: No packages marked for reinstall. 2020-11-16T21:12:14.858 INFO:teuthology.orchestra.run.smithi120:> sudo yum reinstall -y kernel || true 2020-11-16T21:12:15.259 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:03 ago on Mon 16 Nov 2020 09:12:12 PM UTC. 2020-11-16T21:12:15.503 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-16T21:12:15.504 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-16T21:12:15.545 INFO:teuthology.orchestra.run.smithi120.stderr:Error: No packages marked for reinstall. 2020-11-16T21:12:15.595 INFO:teuthology.orchestra.run.smithi120:> rpm -q kernel --last | head -n 1 2020-11-16T21:12:15.661 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-4.18.0-193.28.1.el8_2.x86_64 Sun 15 Nov 2020 12:12:26 PM UTC 2020-11-16T21:12:15.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-193.28.1.el8_2.x86_64 2020-11-16T21:12:15.663 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-16T21:12:15.663 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-11-16T21:12:15.663 INFO:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-16T21:12:15.701 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2020-11-16T21:12:15.715 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-16T21:12:15.725 INFO:teuthology.task.internal:Creating test directory... 2020-11-16T21:12:15.726 INFO:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-16T21:12:15.769 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-16T21:12:15.775 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-16T21:12:15.822 INFO:teuthology.task.internal:Creating archive directory... 2020-11-16T21:12:15.823 INFO:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-16T21:12:15.843 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-16T21:12:15.852 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-16T21:12:15.853 INFO:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-16T21:12:15.926 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:12:15.939 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:12:15.940 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-16T21:12:15.949 INFO:teuthology.task.internal:Configuring sudo... 2020-11-16T21:12:15.949 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-16T21:12:16.008 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-16T21:12:16.017 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-16T21:12:16.018 INFO:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-16T21:12:16.061 INFO:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T21:12:16.135 INFO:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T21:12:16.198 INFO:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T21:12:16.271 INFO:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T21:12:16.319 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:12:16.320 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T21:12:16.371 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T21:12:16.384 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:12:16.384 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:12:16.384 INFO:teuthology.orchestra.run.smithi120.stderr:152 bytes copied, 2.9802e-05 s, 5.1 MB/s 2020-11-16T21:12:16.385 INFO:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2020-11-16T21:12:16.502 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-16T21:12:16.832 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-16T21:12:25.214 INFO:teuthology.task.internal:Starting timer... 2020-11-16T21:12:25.215 INFO:teuthology.run_tasks:Running task pcp... 2020-11-16T21:12:25.229 INFO:teuthology.run_tasks:Running task selinux... 2020-11-16T21:12:25.252 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-11-16T21:12:25.252 INFO:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2020-11-16T21:12:25.332 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: [ OK ] 2020-11-16T21:12:25.333 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-16T21:12:25.334 INFO:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2020-11-16T21:12:25.354 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2020-11-16T21:12:25.355 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2020-11-16T21:12:25.355 INFO:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-16T21:12:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-16T21:12:25.425 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-16T21:12:25.426 INFO:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2020-11-16T21:12:25.494 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-16T21:12:25.503 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-16T21:12:29.616 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-16T21:12:29.662 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-16T21:12:29.675 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-16T21:12:29.676 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-16T21:12:34.109 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-16T21:14:06.312 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-16T21:14:16.829 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-16T21:14:16.848 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:14:16 +0000 (0:00:04.184) 0:00:04.184 ******* 2020-11-16T21:16:31.498 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi120.front.sepia.ceph.com] 2020-11-16T21:16:31.565 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:31 +0000 (0:02:14.717) 0:02:18.901 ******* 2020-11-16T21:16:31.880 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-16T21:16:31.882 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-16T21:16:31.910 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:31 +0000 (0:00:00.345) 0:02:19.246 ******* 2020-11-16T21:16:36.764 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-16T21:16:36.766 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-16T21:16:36.797 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:36 +0000 (0:00:04.886) 0:02:24.133 ******* 2020-11-16T21:16:36.824 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-16T21:16:36.826 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:16:36.855 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:36 +0000 (0:00:00.057) 0:02:24.191 ******* 2020-11-16T21:16:37.151 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-16T21:16:37.153 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T21:16:37.184 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:37 +0000 (0:00:00.329) 0:02:24.520 ******* 2020-11-16T21:16:54.964 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-16T21:16:54.966 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605561397.1984007-104750742818118/source", "state": "file", "uid": 0} 2020-11-16T21:16:54.996 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:54 +0000 (0:00:17.811) 0:02:42.332 ******* 2020-11-16T21:16:55.496 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-16T21:16:55.498 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-16T21:16:55.500 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-16T21:16:55.582 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-16T21:16:55.588 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-16T21:16:55.715 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:55 +0000 (0:00:00.717) 0:02:43.050 ******* 2020-11-16T21:16:56.337 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi120.front.sepia.ceph.com] 2020-11-16T21:16:56.408 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-16T21:16:56.410 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:16:56 +0000 (0:00:00.694) 0:02:43.744 ******* 2020-11-16T21:17:01.456 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-16T21:17:01.458 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.171911", "end": "2020-11-16 21:17:01.121275", "rc": 0, "start": "2020-11-16 21:17:00.949364", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-16T21:17:01.489 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:01 +0000 (0:00:05.080) 0:02:48.825 ******* 2020-11-16T21:17:10.580 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-16T21:17:10.581 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.074675", "end": "2020-11-16 21:17:10.056871", "failed_when_result": false, "rc": 0, "start": "2020-11-16 21:17:09.982196", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-16T21:17:10.612 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:10 +0000 (0:00:09.122) 0:02:57.948 ******* 2020-11-16T21:17:44.091 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-16T21:17:44.093 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-16T21:17:44.124 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:44 +0000 (0:00:33.512) 0:03:31.460 ******* 2020-11-16T21:17:44.151 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-16T21:17:44.153 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:44.183 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:44 +0000 (0:00:00.059) 0:03:31.519 ******* 2020-11-16T21:17:52.589 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-16T21:17:52.591 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.620 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:52 +0000 (0:00:08.436) 0:03:39.956 ******* 2020-11-16T21:17:52.647 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-16T21:17:52.648 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.677 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:52 +0000 (0:00:00.057) 0:03:40.013 ******* 2020-11-16T21:17:52.703 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-16T21:17:52.705 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.734 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:52 +0000 (0:00:00.056) 0:03:40.070 ******* 2020-11-16T21:17:52.760 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-16T21:17:52.761 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.790 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:52 +0000 (0:00:00.056) 0:03:40.126 ******* 2020-11-16T21:17:52.838 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-16T21:17:52.839 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.849 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.851 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd 2020-11-16T21:17:52.852 INFO:teuthology.task.ansible.out:0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.860 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.862 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.871 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.873 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.882 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.884 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.894 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.896 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.905 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.907 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.916 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.918 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.927 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.929 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.939 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.940 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.950 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.951 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.961 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.962 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.972 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.974 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.983 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.985 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.995 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:52.997 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.006 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.008 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.017 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.019 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.029 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.031 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.040 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.041 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.051 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.052 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.062 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.064 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.073 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.075 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.084 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.086 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.096 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.097 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.107 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.110 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.118 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.120 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.130 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.132 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.141 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.143 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.153 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.154 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.164 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.166 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.175 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.177 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.186 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.188 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.198 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.199 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.209 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.210 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.220 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.221 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.231 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.233 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.242 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.243 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.258 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.265 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.266 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.276 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.277 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.278 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.288 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.290 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.300 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.301 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.311 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.313 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.323 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.325 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.334 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.336 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.345 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.347 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.357 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.359 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.368 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.370 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.380 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.381 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.391 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.392 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.402 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.403 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.413 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.415 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.424 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.426 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.435 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.437 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.447 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.448 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.458 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.459 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.459 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.470 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.472 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.481 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.482 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dorinda', 'ovpn': 'dorinda@ideapad /BLvrh5FMYG1lXzwak5Z+w d028380a40a95f162324dfbb024cfa98b5771a1b90741fbfc908cad93ef09a14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dorinda", "ovpn": "dorinda@ideapad /BLvrh5FMYG1lXzwak5Z+w d028380a40a95f162324dfbb024cfa98b5771a1b90741fbfc908cad93ef09a14"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.492 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:17:53.538 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:17:53 +0000 (0:00:00.747) 0:03:40.874 ******* 2020-11-16T21:18:05.903 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-16T21:18:05.936 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:05 +0000 (0:00:12.398) 0:03:53.272 ******* 2020-11-16T21:18:05.961 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-16T21:18:05.963 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:05.997 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:05 +0000 (0:00:00.060) 0:03:53.333 ******* 2020-11-16T21:18:06.023 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-16T21:18:06.024 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.054 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:06 +0000 (0:00:00.057) 0:03:53.390 ******* 2020-11-16T21:18:06.080 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-16T21:18:06.082 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.111 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:06 +0000 (0:00:00.056) 0:03:53.447 ******* 2020-11-16T21:18:06.136 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-16T21:18:06.138 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.168 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:06 +0000 (0:00:00.057) 0:03:53.504 ******* 2020-11-16T21:18:06.194 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-11-16T21:18:06.196 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.225 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:06 +0000 (0:00:00.056) 0:03:53.561 ******* 2020-11-16T21:18:06.261 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-16T21:18:06.262 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.273 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.274 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.275 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.285 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.286 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.288 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.297 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.298 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.300 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.310 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.311 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.322 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.323 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.324 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.334 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.336 INFO:teuthology.task.ansible.out: skipping: [smithi120.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.346 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.347 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.348 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.358 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.359 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.361 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.372 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.372 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.383 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.384 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.386 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.396 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.397 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.398 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.408 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.409 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.410 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.420 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.421 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.423 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.433 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.433 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.435 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.445 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.446 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.447 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.457 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-16T21:18:06.492 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:06 +0000 (0:00:00.267) 0:03:53.828 ******* 2020-11-16T21:18:19.821 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-16T21:18:19.823 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-11-16T21:18:19.854 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:19 +0000 (0:00:13.362) 0:04:07.190 ******* 2020-11-16T21:18:19.881 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-16T21:18:19.883 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => { "msg": "Host smithi120.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-11-16T21:18:19.915 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:19 +0000 (0:00:00.060) 0:04:07.251 ******* 2020-11-16T21:18:19.952 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-16T21:18:19.954 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-11-16T21:18:19.984 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:19 +0000 (0:00:00.069) 0:04:07.320 ******* 2020-11-16T21:18:20.011 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-16T21:18:20.013 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:20.045 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.057) 0:04:07.378 ******* 2020-11-16T21:18:20.605 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-16T21:18:20.607 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.363506", "end": "2020-11-16 21:18:20.587205", "rc": 0, "start": "2020-11-16 21:18:20.223699", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-16T21:18:20.637 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.594) 0:04:07.972 ******* 2020-11-16T21:18:20.663 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-16T21:18:20.665 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:20.696 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.059) 0:04:08.032 ******* 2020-11-16T21:18:20.723 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-16T21:18:20.724 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:20.754 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.058) 0:04:08.090 ******* 2020-11-16T21:18:20.782 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-16T21:18:20.784 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:20.812 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.058) 0:04:08.148 ******* 2020-11-16T21:18:20.838 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-16T21:18:20.840 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:20.870 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.057) 0:04:08.206 ******* 2020-11-16T21:18:20.896 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-16T21:18:20.898 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:20.927 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.057) 0:04:08.263 ******* 2020-11-16T21:18:20.954 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-16T21:18:20.955 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:20.985 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:20 +0000 (0:00:00.057) 0:04:08.321 ******* 2020-11-16T21:18:21.021 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-16T21:18:21.023 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T21:18:21.053 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.068) 0:04:08.389 ******* 2020-11-16T21:18:21.080 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-16T21:18:21.081 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:21.112 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.059) 0:04:08.448 ******* 2020-11-16T21:18:21.138 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-16T21:18:21.140 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:21.169 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.057) 0:04:08.505 ******* 2020-11-16T21:18:21.196 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-16T21:18:21.198 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T21:18:21.226 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.057) 0:04:08.563 ******* 2020-11-16T21:18:21.252 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-16T21:18:21.254 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:21.283 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.056) 0:04:08.619 ******* 2020-11-16T21:18:21.319 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-16T21:18:21.321 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T21:18:21.353 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.070) 0:04:08.689 ******* 2020-11-16T21:18:21.380 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-16T21:18:21.382 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:21.411 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.058) 0:04:08.747 ******* 2020-11-16T21:18:21.438 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-16T21:18:21.440 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:21.470 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.058) 0:04:08.806 ******* 2020-11-16T21:18:21.497 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-16T21:18:21.499 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:21.528 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.058) 0:04:08.864 ******* 2020-11-16T21:18:21.558 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-16T21:18:21.560 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:21.602 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:21 +0000 (0:00:00.073) 0:04:08.938 ******* 2020-11-16T21:18:25.749 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T21:18:25.780 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:25 +0000 (0:00:04.177) 0:04:13.116 ******* 2020-11-16T21:18:25.805 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-16T21:18:25.806 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:25.836 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:25 +0000 (0:00:00.056) 0:04:13.172 ******* 2020-11-16T21:18:25.863 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-16T21:18:25.865 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:25.894 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:25 +0000 (0:00:00.058) 0:04:13.230 ******* 2020-11-16T21:18:25.921 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-16T21:18:25.922 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:25.952 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:25 +0000 (0:00:00.057) 0:04:13.288 ******* 2020-11-16T21:18:25.979 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-16T21:18:25.980 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:26.009 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:26 +0000 (0:00:00.057) 0:04:13.345 ******* 2020-11-16T21:18:26.035 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-16T21:18:26.036 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:26.065 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:26 +0000 (0:00:00.055) 0:04:13.401 ******* 2020-11-16T21:18:30.201 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-16T21:18:30.203 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:30.233 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:30 +0000 (0:00:04.168) 0:04:17.569 ******* 2020-11-16T21:18:30.259 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-16T21:18:30.261 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T21:18:30.290 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:30 +0000 (0:00:00.057) 0:04:17.627 ******* 2020-11-16T21:18:30.317 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-16T21:18:30.319 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:30.348 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:30 +0000 (0:00:00.057) 0:04:17.684 ******* 2020-11-16T21:18:30.384 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-16T21:18:30.385 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:30.417 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:30 +0000 (0:00:00.069) 0:04:17.753 ******* 2020-11-16T21:18:30.628 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-16T21:18:30.630 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T21:18:30.660 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:30 +0000 (0:00:00.242) 0:04:17.996 ******* 2020-11-16T21:18:31.130 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-16T21:18:31.131 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-11-16T21:18:31.568 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-11-16T21:18:31.602 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:31 +0000 (0:00:00.942) 0:04:18.938 ******* 2020-11-16T21:18:32.079 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-16T21:18:32.081 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-11-16T21:18:53.347 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-11-16T21:18:53.379 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:53 +0000 (0:00:21.777) 0:04:40.715 ******* 2020-11-16T21:18:53.406 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-16T21:18:53.407 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:53.437 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:53 +0000 (0:00:00.057) 0:04:40.773 ******* 2020-11-16T21:18:53.464 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-16T21:18:53.466 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:53.495 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:53 +0000 (0:00:00.057) 0:04:40.831 ******* 2020-11-16T21:18:53.522 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:53.553 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:53 +0000 (0:00:00.058) 0:04:40.889 ******* 2020-11-16T21:18:53.580 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-16T21:18:53.582 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:53.611 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:53 +0000 (0:00:00.058) 0:04:40.947 ******* 2020-11-16T21:18:53.638 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-16T21:18:53.640 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:53.670 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:53 +0000 (0:00:00.059) 0:04:41.006 ******* 2020-11-16T21:18:53.697 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-16T21:18:53.698 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:53.728 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:53 +0000 (0:00:00.057) 0:04:41.064 ******* 2020-11-16T21:18:58.021 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-16T21:18:58.022 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:58.052 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:58 +0000 (0:00:04.323) 0:04:45.388 ******* 2020-11-16T21:18:58.070 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-16T21:18:58.072 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:58.101 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:58 +0000 (0:00:00.048) 0:04:45.437 ******* 2020-11-16T21:18:58.127 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-16T21:18:58.129 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:18:58.159 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:18:58 +0000 (0:00:00.057) 0:04:45.495 ******* 2020-11-16T21:19:15.514 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-16T21:19:15.516 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.389740", "end": "2020-11-16 21:19:15.500331", "rc": 0, "start": "2020-11-16 21:19:15.110591", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:19:15.554 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:15 +0000 (0:00:17.395) 0:05:02.890 ******* 2020-11-16T21:19:54.315 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-16T21:19:54.317 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-11-16T21:19:54.346 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:54 +0000 (0:00:38.792) 0:05:41.682 ******* 2020-11-16T21:19:54.372 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-16T21:19:54.374 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:19:54.404 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:54 +0000 (0:00:00.057) 0:05:41.740 ******* 2020-11-16T21:19:54.429 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-16T21:19:54.431 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:19:54.461 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:54 +0000 (0:00:00.057) 0:05:41.797 ******* 2020-11-16T21:19:54.486 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-16T21:19:54.488 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:19:54.517 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:54 +0000 (0:00:00.056) 0:05:41.853 ******* 2020-11-16T21:19:54.543 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-16T21:19:54.544 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:19:54.573 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:54 +0000 (0:00:00.056) 0:05:41.909 ******* 2020-11-16T21:19:55.021 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-16T21:19:55.023 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-11-16T21:19:55.052 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:55 +0000 (0:00:00.478) 0:05:42.388 ******* 2020-11-16T21:19:59.779 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-16T21:19:59.781 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools", "Installed: smartmontools-1:7.1-1.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2020-11-16T21:19:59.811 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:19:59 +0000 (0:00:04.759) 0:05:47.147 ******* 2020-11-16T21:20:12.941 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-16T21:20:12.943 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-11-16T21:20:13.405 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-11-16T21:20:13.829 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-11-16T21:20:13.862 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:13 +0000 (0:00:14.050) 0:06:01.198 ******* 2020-11-16T21:20:14.054 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-16T21:20:14.056 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-16T21:20:14.086 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:14 +0000 (0:00:00.224) 0:06:01.422 ******* 2020-11-16T21:20:18.847 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-16T21:20:18.848 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-11-16T21:20:24.072 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-11-16T21:20:24.073 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-11-16T21:20:24.116 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:24 +0000 (0:00:10.029) 0:06:11.452 ******* 2020-11-16T21:20:28.410 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:20:28.442 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:28 +0000 (0:00:04.325) 0:06:15.778 ******* 2020-11-16T21:20:28.479 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-16T21:20:28.480 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-11-16T21:20:28.511 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:28 +0000 (0:00:00.068) 0:06:15.846 ******* 2020-11-16T21:20:28.712 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-16T21:20:28.714 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-11-16T21:20:28.743 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:28 +0000 (0:00:00.232) 0:06:16.079 ******* 2020-11-16T21:20:28.780 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-16T21:20:28.782 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T21:20:28.791 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T21:20:28.802 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T21:20:28.812 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T21:20:28.844 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:28 +0000 (0:00:00.100) 0:06:16.180 ******* 2020-11-16T21:20:33.698 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-16T21:20:33.700 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-11-16T21:20:33.730 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:33 +0000 (0:00:04.886) 0:06:21.066 ******* 2020-11-16T21:20:33.757 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-16T21:20:33.758 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:20:33.788 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:33 +0000 (0:00:00.057) 0:06:21.124 ******* 2020-11-16T21:20:38.748 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-16T21:20:38.750 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-11-16T21:20:38.786 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:38 +0000 (0:00:04.998) 0:06:26.122 ******* 2020-11-16T21:20:39.258 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-16T21:20:39.260 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-11-16T21:20:39.290 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:39 +0000 (0:00:00.503) 0:06:26.626 ******* 2020-11-16T21:20:39.328 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-16T21:20:39.330 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-11-16T21:20:39.362 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:39 +0000 (0:00:00.071) 0:06:26.698 ******* 2020-11-16T21:20:43.824 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-16T21:20:43.826 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T21:20:43.856 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:43 +0000 (0:00:04.494) 0:06:31.192 ******* 2020-11-16T21:20:44.099 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-16T21:20:44.101 INFO:teuthology.task.ansible.out:fatal: [smithi120.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-11-16T21:20:44.131 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:44 +0000 (0:00:00.275) 0:06:31.467 ******* 2020-11-16T21:20:44.159 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-16T21:20:44.160 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:20:44.190 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:44 +0000 (0:00:00.059) 0:06:31.526 ******* 2020-11-16T21:20:49.495 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-16T21:20:49.497 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2020-11-16T21:20:49.536 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:49 +0000 (0:00:05.345) 0:06:36.872 ******* 2020-11-16T21:20:58.584 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-16T21:20:58.585 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-16 20:21:24 UTC", "ActiveEnterTimestampMonotonic": "15178699", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount basic.target local-fs.target var-run.mount network.target time-sync.target systemd-journald.socket system.slice -.mount nss-lookup.target sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-16 20:21:24 UTC", "AssertTimestampMonotonic": "15177955", "Before": "xdm.service shutdown.target multi-user.target getty@tty1.service plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-16 20:21:24 UTC", "ConditionTimestampMonotonic": "15177953", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-11-16T21:20:58.586 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2511", "ExecMainStartTimestamp": "Mon 2020-11-16 20:21:24 UTC", "ExecMainStartTimestampMonotonic": "15178670", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-16 20:21:24 UTC", "InactiveExitTimestampMonotonic": "15178699", "InvocationID": "9d36f525299e4fa09599411efcf071bf", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitD 2020-11-16T21:20:58.588 INFO:teuthology.task.ansible.out:ATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127425", "LimitNPROCSoft": "127425", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127425", "LimitSIGPENDINGSoft": "127425", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2511", "MemoryAccounting": "yes", "MemoryCurrent": "1978368", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode 2020-11-16T21:20:58.589 INFO:teuthology.task.ansible.out:": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-16 20:21:24 UTC", "StateChangeTimestampMonotonic": "15178699", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-16 20:21:24 UTC", "WatchdogTimestampMonotonic": "15178698", "WatchdogUSec": "0"}} 2020-11-16T21:20:58.620 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:58 +0000 (0:00:09.083) 0:06:45.955 ******* 2020-11-16T21:20:58.814 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-16T21:20:58.816 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001734", "end": "2020-11-16 21:20:58.799109", "rc": 0, "start": "2020-11-16 21:20:58.797375", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-11-16T21:20:58.847 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:20:58 +0000 (0:00:00.227) 0:06:46.183 ******* 2020-11-16T21:21:03.969 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-16T21:21:03.971 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-11-16T21:21:04.001 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:21:03 +0000 (0:00:05.154) 0:06:51.337 ******* 2020-11-16T21:21:09.028 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-16T21:21:09.030 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-11-16T21:21:09.060 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:21:09 +0000 (0:00:05.058) 0:06:56.396 ******* 2020-11-16T21:21:19.056 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-16T21:21:19.058 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-11-16T21:21:19.088 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:21:19 +0000 (0:00:10.027) 0:07:06.423 ******* 2020-11-16T21:21:35.446 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-16T21:21:35.448 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.155252", "end": "2020-11-16 21:21:35.431683", "failed_when_result": false, "rc": 0, "start": "2020-11-16 21:21:19.276431", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-11-16T21:21:35.478 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:21:35 +0000 (0:00:16.390) 0:07:22.814 ******* 2020-11-16T21:22:50.850 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-16T21:22:50.852 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-11-16T21:22:50.883 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:22:50 +0000 (0:01:15.405) 0:08:38.219 ******* 2020-11-16T21:22:51.181 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-16T21:22:51.183 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.097345", "end": "2020-11-16 21:22:51.172613", "rc": 0, "start": "2020-11-16 21:22:51.075268", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:22:51.213 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:22:51 +0000 (0:00:00.329) 0:08:38.549 ******* 2020-11-16T21:22:51.416 INFO:teuthology.task.ansible.out: 2020-11-16T21:22:51.417 INFO:teuthology.task.ansible.out:TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009444", "end": "2020-11-16 21:22:51.407223", "rc": 0, "start": "2020-11-16 21:22:51.397779", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:22:51.450 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:22:51 +0000 (0:00:00.236) 0:08:38.786 ******* 2020-11-16T21:23:10.404 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-16T21:23:10.407 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.538319", "end": "2020-11-16 21:23:08.170591", "rc": 0, "start": "2020-11-16 21:22:51.632272", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:23:10.436 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:10 +0000 (0:00:18.986) 0:08:57.772 ******* 2020-11-16T21:23:18.892 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-16T21:23:18.894 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-11-16T21:23:18.930 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:18 +0000 (0:00:08.493) 0:09:06.266 ******* 2020-11-16T21:23:18.958 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-16T21:23:18.960 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-11-16T21:23:18.991 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:18 +0000 (0:00:00.060) 0:09:06.327 ******* 2020-11-16T21:23:19.030 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-16T21:23:19.031 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-16T21:23:19.061 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:19 +0000 (0:00:00.070) 0:09:06.397 ******* 2020-11-16T21:23:19.109 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-16T21:23:19.110 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-11-16T21:23:19.144 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:19 +0000 (0:00:00.082) 0:09:06.480 ******* 2020-11-16T21:23:19.185 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-16T21:23:19.187 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-16T21:23:19.218 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:19 +0000 (0:00:00.073) 0:09:06.554 ******* 2020-11-16T21:23:23.561 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-16T21:23:23.563 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-16T21:23:23.593 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:23 +0000 (0:00:04.375) 0:09:10.929 ******* 2020-11-16T21:23:23.798 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-16T21:23:23.831 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:23 +0000 (0:00:00.236) 0:09:11.165 ******* 2020-11-16T21:23:23.858 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-16T21:23:23.860 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:23:23.890 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:23 +0000 (0:00:00.060) 0:09:11.226 ******* 2020-11-16T21:23:28.018 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-16T21:23:28.020 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:23:28.050 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:28 +0000 (0:00:04.159) 0:09:15.386 ******* 2020-11-16T21:23:28.275 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi120.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-11-16T21:23:28.308 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:28 +0000 (0:00:00.258) 0:09:15.644 ******* 2020-11-16T21:23:36.876 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-16T21:23:36.878 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-11-16T21:23:36.918 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:36 +0000 (0:00:08.609) 0:09:24.254 ******* 2020-11-16T21:23:36.987 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-16T21:23:36.989 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi120"}, "changed": false} 2020-11-16T21:23:37.040 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:37 +0000 (0:00:00.121) 0:09:24.375 ******* 2020-11-16T21:23:46.327 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-16T21:23:46.329 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi120", "ansible_hostname": "smithi120", "ansible_nodename": "smithi120"}, "changed": false, "name": "smithi120"} 2020-11-16T21:23:46.357 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:46 +0000 (0:00:09.317) 0:09:33.693 ******* 2020-11-16T21:23:55.180 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-16T21:23:55.182 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-11-16T21:23:55.234 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:23:55 +0000 (0:00:08.876) 0:09:42.570 ******* 2020-11-16T21:24:15.861 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-16T21:24:15.863 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-11-16T21:24:15.893 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:24:15 +0000 (0:00:20.658) 0:10:03.229 ******* 2020-11-16T21:24:16.640 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-16T21:24:16.641 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-16T21:24:16.673 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:24:16 +0000 (0:00:00.779) 0:10:04.009 ******* 2020-11-16T21:24:25.231 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-16T21:24:25.233 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-11-16T21:24:25.263 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:24:25 +0000 (0:00:08.589) 0:10:12.599 ******* 2020-11-16T21:24:25.458 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-16T21:24:25.460 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-16T21:24:25.507 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:24:25 +0000 (0:00:00.243) 0:10:12.843 ******* 2020-11-16T21:24:41.885 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-16T21:24:41.886 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-11-16T21:24:41.916 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:24:41 +0000 (0:00:16.409) 0:10:29.252 ******* 2020-11-16T21:24:42.135 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-16T21:24:42.137 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-16T21:24:42.183 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:24:42 +0000 (0:00:00.266) 0:10:29.519 ******* 2020-11-16T21:25:02.889 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-11-16T21:25:02.922 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:02 +0000 (0:00:20.739) 0:10:50.258 ******* 2020-11-16T21:25:07.444 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-16T21:25:07.446 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-11-16T21:25:07.477 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:07 +0000 (0:00:04.555) 0:10:54.813 ******* 2020-11-16T21:25:13.442 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-16T21:25:13.443 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.688333", "end": "2020-11-16 21:25:13.429337", "rc": 0, "start": "2020-11-16 21:25:11.741004", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:25:13.474 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:13 +0000 (0:00:05.996) 0:11:00.810 ******* 2020-11-16T21:25:26.110 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-16T21:25:26.112 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2020-11-16T21:25:26.543 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2020-11-16T21:25:26.995 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2020-11-16T21:25:27.030 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:27 +0000 (0:00:13.555) 0:11:14.365 ******* 2020-11-16T21:25:27.511 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-16T21:25:27.512 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-11-16T21:25:27.546 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:27 +0000 (0:00:00.516) 0:11:14.882 ******* 2020-11-16T21:25:28.028 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-16T21:25:28.030 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-11-16T21:25:40.535 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-11-16T21:25:40.959 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-11-16T21:25:40.994 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:40 +0000 (0:00:13.447) 0:11:28.330 ******* 2020-11-16T21:25:45.110 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-16T21:25:45.142 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:45 +0000 (0:00:04.148) 0:11:32.478 ******* 2020-11-16T21:25:49.253 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:49.290 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:04.147) 0:11:36.626 ******* 2020-11-16T21:25:49.328 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-16T21:25:49.330 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-16T21:25:49.339 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-16T21:25:49.374 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:00.082) 0:11:36.708 ******* 2020-11-16T21:25:49.399 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-16T21:25:49.401 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:49.433 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:00.060) 0:11:36.769 ******* 2020-11-16T21:25:49.472 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:25:49.474 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-11-16T21:25:49.504 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:00.070) 0:11:36.840 ******* 2020-11-16T21:25:49.530 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-16T21:25:49.532 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:49.565 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:00.060) 0:11:36.901 ******* 2020-11-16T21:25:49.607 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-16T21:25:49.608 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:49.640 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:00.075) 0:11:36.976 ******* 2020-11-16T21:25:49.875 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-16T21:25:49.877 INFO:teuthology.task.ansible.out:fatal: [smithi120.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006854", "end": "2020-11-16 21:25:49.857592", "msg": "non-zero return code", "rc": 1, "start": "2020-11-16 21:25:49.850738", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-11-16T21:25:49.909 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:00.268) 0:11:37.244 ******* 2020-11-16T21:25:49.936 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-16T21:25:49.938 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:49.968 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:49 +0000 (0:00:00.059) 0:11:37.304 ******* 2020-11-16T21:25:52.019 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-16T21:25:52.022 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-11-16T21:25:52.051 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:52 +0000 (0:00:02.083) 0:11:39.387 ******* 2020-11-16T21:25:53.963 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-16T21:25:53.965 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2020-11-16T21:25:53.995 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:53 +0000 (0:00:01.943) 0:11:41.331 ******* 2020-11-16T21:25:56.887 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-16T21:25:56.889 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64"]} 2020-11-16T21:25:56.920 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:56 +0000 (0:00:02.924) 0:11:44.255 ******* 2020-11-16T21:25:56.953 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-16T21:25:56.955 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:56.988 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:56 +0000 (0:00:00.068) 0:11:44.324 ******* 2020-11-16T21:25:57.914 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-16T21:25:57.915 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-11-16T21:25:57.954 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:57 +0000 (0:00:00.966) 0:11:45.290 ******* 2020-11-16T21:25:57.983 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-16T21:25:57.985 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:58.021 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:58 +0000 (0:00:00.066) 0:11:45.357 ******* 2020-11-16T21:25:58.320 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-16T21:25:58.321 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket dbus.service basic.target polkit.service system.slice sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ip6tables.service iptables.service shutdown.target ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-11-16T21:25:58.322 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127425", "LimitNPROCSoft": "127425", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127425", 2020-11-16T21:25:58.324 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127425", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-11-16T21:25:58.325 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-11-16T21:25:58.354 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:58 +0000 (0:00:00.332) 0:11:45.690 ******* 2020-11-16T21:25:58.637 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-16T21:25:58.639 INFO:teuthology.task.ansible.out:fatal: [smithi120.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-11-16T21:25:58.670 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:58 +0000 (0:00:00.316) 0:11:46.006 ******* 2020-11-16T21:25:59.366 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-16T21:25:59.368 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-11-16T21:25:59.398 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.727) 0:11:46.734 ******* 2020-11-16T21:25:59.425 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-16T21:25:59.426 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.458 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.060) 0:11:46.794 ******* 2020-11-16T21:25:59.485 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-16T21:25:59.487 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.517 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.058) 0:11:46.853 ******* 2020-11-16T21:25:59.544 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-16T21:25:59.546 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.576 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.058) 0:11:46.912 ******* 2020-11-16T21:25:59.603 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-16T21:25:59.604 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.634 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.058) 0:11:46.970 ******* 2020-11-16T21:25:59.663 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.694 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.059) 0:11:47.030 ******* 2020-11-16T21:25:59.721 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-16T21:25:59.723 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.753 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.058) 0:11:47.089 ******* 2020-11-16T21:25:59.780 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-16T21:25:59.811 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.058) 0:11:47.147 ******* 2020-11-16T21:25:59.838 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-16T21:25:59.840 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.870 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.058) 0:11:47.206 ******* 2020-11-16T21:25:59.897 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-16T21:25:59.899 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.928 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.058) 0:11:47.264 ******* 2020-11-16T21:25:59.965 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-16T21:25:59.966 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-16T21:25:59.997 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:25:59 +0000 (0:00:00.068) 0:11:47.333 ******* 2020-11-16T21:26:00.024 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.055 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.058) 0:11:47.391 ******* 2020-11-16T21:26:00.084 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-16T21:26:00.115 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.059) 0:11:47.451 ******* 2020-11-16T21:26:00.142 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-16T21:26:00.143 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.173 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.058) 0:11:47.509 ******* 2020-11-16T21:26:00.200 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-16T21:26:00.202 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.232 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.058) 0:11:47.568 ******* 2020-11-16T21:26:00.259 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.297 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.065) 0:11:47.633 ******* 2020-11-16T21:26:00.325 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.360 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.062) 0:11:47.696 ******* 2020-11-16T21:26:00.387 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-16T21:26:00.389 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.419 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.059) 0:11:47.755 ******* 2020-11-16T21:26:00.452 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.494 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.075) 0:11:47.830 ******* 2020-11-16T21:26:00.523 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-16T21:26:00.525 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.556 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.061) 0:11:47.892 ******* 2020-11-16T21:26:00.584 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.618 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.062) 0:11:47.954 ******* 2020-11-16T21:26:00.646 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-16T21:26:00.647 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.685 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.066) 0:11:48.021 ******* 2020-11-16T21:26:00.712 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.745 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.059) 0:11:48.080 ******* 2020-11-16T21:26:00.771 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-16T21:26:00.773 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.811 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.066) 0:11:48.147 ******* 2020-11-16T21:26:00.842 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-16T21:26:00.844 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.877 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.065) 0:11:48.213 ******* 2020-11-16T21:26:00.905 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.937 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.059) 0:11:48.273 ******* 2020-11-16T21:26:00.964 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-16T21:26:00.966 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:00.997 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:00 +0000 (0:00:00.059) 0:11:48.333 ******* 2020-11-16T21:26:01.023 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-16T21:26:01.025 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.057 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.059) 0:11:48.392 ******* 2020-11-16T21:26:01.083 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-16T21:26:01.085 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.116 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.059) 0:11:48.452 ******* 2020-11-16T21:26:01.143 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-16T21:26:01.144 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.174 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.058) 0:11:48.510 ******* 2020-11-16T21:26:01.215 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-16T21:26:01.217 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.246 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.071) 0:11:48.582 ******* 2020-11-16T21:26:01.272 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-16T21:26:01.274 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.304 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.058) 0:11:48.640 ******* 2020-11-16T21:26:01.331 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.363 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.058) 0:11:48.699 ******* 2020-11-16T21:26:01.389 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-16T21:26:01.391 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.421 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.057) 0:11:48.757 ******* 2020-11-16T21:26:01.447 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-16T21:26:01.449 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.479 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.058) 0:11:48.815 ******* 2020-11-16T21:26:01.506 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:26:01.508 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.539 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.059) 0:11:48.875 ******* 2020-11-16T21:26:01.567 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-16T21:26:01.569 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.606 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.067) 0:11:48.942 ******* 2020-11-16T21:26:01.636 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.672 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.065) 0:11:49.007 ******* 2020-11-16T21:26:01.701 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-16T21:26:01.703 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.733 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.061) 0:11:49.069 ******* 2020-11-16T21:26:01.760 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-16T21:26:01.761 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:01.793 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:01 +0000 (0:00:00.059) 0:11:49.128 ******* 2020-11-16T21:26:02.007 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-16T21:26:02.009 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-11-16T21:26:02.041 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.248) 0:11:49.377 ******* 2020-11-16T21:26:02.247 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-16T21:26:02.249 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-16T21:26:02.279 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.238) 0:11:49.615 ******* 2020-11-16T21:26:02.472 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-16T21:26:02.473 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-11-16T21:26:02.506 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.226) 0:11:49.842 ******* 2020-11-16T21:26:02.533 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.565 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.058) 0:11:49.901 ******* 2020-11-16T21:26:02.592 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-16T21:26:02.593 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.624 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.058) 0:11:49.960 ******* 2020-11-16T21:26:02.650 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-16T21:26:02.652 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.682 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.057) 0:11:50.018 ******* 2020-11-16T21:26:02.709 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-16T21:26:02.711 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.744 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.061) 0:11:50.080 ******* 2020-11-16T21:26:02.770 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-16T21:26:02.772 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.803 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.059) 0:11:50.139 ******* 2020-11-16T21:26:02.830 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-16T21:26:02.832 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.863 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.059) 0:11:50.199 ******* 2020-11-16T21:26:02.890 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-16T21:26:02.891 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.922 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.058) 0:11:50.257 ******* 2020-11-16T21:26:02.948 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-16T21:26:02.950 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:02.993 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:02 +0000 (0:00:00.069) 0:11:50.327 ******* 2020-11-16T21:26:03.024 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-16T21:26:03.026 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.056 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.064) 0:11:50.392 ******* 2020-11-16T21:26:03.083 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-16T21:26:03.085 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.119 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.063) 0:11:50.455 ******* 2020-11-16T21:26:03.147 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.182 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.062) 0:11:50.518 ******* 2020-11-16T21:26:03.209 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-16T21:26:03.211 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.242 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.059) 0:11:50.578 ******* 2020-11-16T21:26:03.268 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-16T21:26:03.270 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.301 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.058) 0:11:50.637 ******* 2020-11-16T21:26:03.329 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-16T21:26:03.331 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.378 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.077) 0:11:50.714 ******* 2020-11-16T21:26:03.414 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.444 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.066) 0:11:50.780 ******* 2020-11-16T21:26:03.471 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-16T21:26:03.473 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.503 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.058) 0:11:50.839 ******* 2020-11-16T21:26:03.530 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-16T21:26:03.531 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.562 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.058) 0:11:50.898 ******* 2020-11-16T21:26:03.589 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-16T21:26:03.591 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.622 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.060) 0:11:50.958 ******* 2020-11-16T21:26:03.649 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-16T21:26:03.651 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.684 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.061) 0:11:51.020 ******* 2020-11-16T21:26:03.711 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-16T21:26:03.713 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.743 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.058) 0:11:51.079 ******* 2020-11-16T21:26:03.769 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-16T21:26:03.771 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.802 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.059) 0:11:51.138 ******* 2020-11-16T21:26:03.829 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-16T21:26:03.831 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.862 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.059) 0:11:51.198 ******* 2020-11-16T21:26:03.889 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:26:03.891 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-16T21:26:03.926 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:03 +0000 (0:00:00.064) 0:11:51.262 ******* 2020-11-16T21:26:03.974 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-16T21:26:03.976 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:03.985 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-16T21:26:04.007 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.018 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.020 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.029 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.065 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:04 +0000 (0:00:00.138) 0:11:51.401 ******* 2020-11-16T21:26:04.261 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-16T21:26:04.262 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003792", "end": "2020-11-16 21:26:04.235808", "rc": 0, "start": "2020-11-16 21:26:04.232016", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-16T21:26:04.295 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:04 +0000 (0:00:00.230) 0:11:51.631 ******* 2020-11-16T21:26:04.322 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-16T21:26:04.324 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.358 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:04 +0000 (0:00:00.061) 0:11:51.693 ******* 2020-11-16T21:26:04.394 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-16T21:26:04.396 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.429 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:04 +0000 (0:00:00.071) 0:11:51.765 ******* 2020-11-16T21:26:04.456 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-16T21:26:04.458 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.488 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:04 +0000 (0:00:00.059) 0:11:51.824 ******* 2020-11-16T21:26:04.515 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-16T21:26:04.516 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:04.546 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:04 +0000 (0:00:00.057) 0:11:51.882 ******* 2020-11-16T21:26:05.483 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-16T21:26:05.484 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-11-16T21:26:05.515 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:05 +0000 (0:00:00.969) 0:11:52.851 ******* 2020-11-16T21:26:05.552 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-16T21:26:05.553 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930529370112, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227180022, 'block_used': 12915086, 'inode_total': 61054976, 'inode_available': 60990124, 'inode_used': 64852, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227180022, "block_size": 4096, "block_total": 240095108, "block_used": 12915086, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990124, "inode_total": 61054976, "inode_used": 64852, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930529370112, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}} 2020-11-16T21:26:05.585 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:05 +0000 (0:00:00.070) 0:11:52.921 ******* 2020-11-16T21:26:05.785 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-16T21:26:05.787 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004974", "end": "2020-11-16 21:26:05.767650", "rc": 0, "start": "2020-11-16 21:26:05.762676", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:05.817 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:05 +0000 (0:00:00.231) 0:11:53.153 ******* 2020-11-16T21:26:05.855 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-16T21:26:05.856 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930529370112, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227180022, 'block_used': 12915086, 'inode_total': 61054976, 'inode_available': 60990124, 'inode_used': 64852, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227180022, "block_size": 4096, "block_total": 240095108, "block_used": 12915086, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990124, "inode_total": 61054976, "inode_used": 64852, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930529370112, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:05.891 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:05 +0000 (0:00:00.074) 0:11:53.227 ******* 2020-11-16T21:26:05.919 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-16T21:26:05.950 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:05 +0000 (0:00:00.058) 0:11:53.285 ******* 2020-11-16T21:26:07.560 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-16T21:26:07.562 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4I64roQTJHIP7UfMaX10YxAV3eec3K8mwV'], 'uuids': ['e0a0c422-55db-4b7b-a100-baa4880c1c4c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.389691", "end": "2020-11-16 21:26:07.539954", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4I64roQTJHIP7UfMaX10YxAV3eec3K8mwV"], "labels": [], "masters": [], "uuids": ["e0a0c422-55db-4b7b-a100-baa4880c1c4c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:06.150263", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T21:26:08.817 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-dwZ050-daVf-TO7M-Y2jG-9eLZ-X8Dv-5oVM2Q', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN', 'nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.067397", "end": "2020-11-16 21:26:08.800469", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-dwZ050-daVf-TO7M-Y2jG-9eLZ-X8Dv-5oVM2Q", "nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN", "nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:07.733072", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-16T21:26:08.819 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T21:26:10.223 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IVMICqeEJd96nrzYQqI6an0Dv22ZhGfrR'], 'uuids': ['3b17c722-b3d4-4e04-99bc-275bb506ec51'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.212622", "end": "2020-11-16 21:26:10.203793", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IVMICqeEJd96nrzYQqI6an0Dv22ZhGfrR"], "labels": [], "masters": [], "uuids": ["3b17c722-b3d4-4e04-99bc-275bb506ec51"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:08.991171", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T21:26:11.467 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IUyD5aHEeCfQLdnNzQll4zxQbcFoE7uP8'], 'uuids': ['a58bbf76-2856-4671-9910-99c9f398d408'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.055000", "end": "2020-11-16 21:26:11.447448", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IUyD5aHEeCfQLdnNzQll4zxQbcFoE7uP8"], "labels": [], "masters": [], "uuids": ["a58bbf76-2856-4671-9910-99c9f398d408"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:10.392448", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T21:26:12.713 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IHO83mNjuenH3YCxAI5rzh16uTpPXouiR'], 'uuids': ['b5238c2d-384f-4e1b-b48d-08ba55d87ea0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.062016", "end": "2020-11-16 21:26:12.697073", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IHO83mNjuenH3YCxAI5rzh16uTpPXouiR"], "labels": [], "masters": [], "uuids": ["b5238c2d-384f-4e1b-b48d-08ba55d87ea0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:11.635057", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T21:26:12.725 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0', 'wwn-0x5000c50091e1af53'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QHP0', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e1af53', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1', 'wwn-0x5000c50091e1af53-part1'], 'uuids': ['9bdf28e7-5b1a-4776-884c-191bd8ad39f5'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0", "wwn-0x5000c50091e1af53"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1", "wwn-0x5000c50091e1af53-part1"], "labels": [], "masters": [], "uuids": ["9bdf28e7-5b1a-4776-884c-191bd8ad39f5"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHP0", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1af53"}}, 2020-11-16T21:26:12.727 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-11-16T21:26:14.134 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IFIHKt9jcRFn0Y2viegiNz3ExVXSERVeK'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.221852", "end": "2020-11-16 21:26:14.113699", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IFIHKt9jcRFn0Y2viegiNz3ExVXSERVeK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:12.891847", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T21:26:14.170 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:14 +0000 (0:00:08.220) 0:12:01.506 ******* 2020-11-16T21:26:14.527 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-16T21:26:14.529 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4I64roQTJHIP7UfMaX10YxAV3eec3K8mwV'], 'uuids': ['e0a0c422-55db-4b7b-a100-baa4880c1c4c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.137535", "end": "2020-11-16 21:26:14.507860", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4I64roQTJHIP7UfMaX10YxAV3eec3K8mwV"], "labels": [], "masters": [], "uuids": ["e0a0c422-55db-4b7b-a100-baa4880c1c4c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:14.370325", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135076 s, 854 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135076 s, 854 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:14.736 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-dwZ050-daVf-TO7M-Y2jG-9eLZ-X8Dv-5oVM2Q', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN', 'nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037225", "end": "2020-11-16 21:26:14.723531", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-dwZ050-daVf-TO7M-Y2jG-9eLZ-X8Dv-5oVM2Q", "nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN", "nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:14.686306", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0346354 s, 3.3 GB/s", "std 2020-11-16T21:26:14.738 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0346354 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:15.081 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IVMICqeEJd96nrzYQqI6an0Dv22ZhGfrR'], 'uuids': ['3b17c722-b3d4-4e04-99bc-275bb506ec51'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149370", "end": "2020-11-16 21:26:15.063333", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IVMICqeEJd96nrzYQqI6an0Dv22ZhGfrR"], "labels": [], "masters": [], "uuids": ["3b17c722-b3d4-4e04-99bc-275bb506ec51"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:14.913963", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146793 s, 786 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146793 s, 786 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:15.402 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IUyD5aHEeCfQLdnNzQll4zxQbcFoE7uP8'], 'uuids': ['a58bbf76-2856-4671-9910-99c9f398d408'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.139456", "end": "2020-11-16 21:26:15.380991", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IUyD5aHEeCfQLdnNzQll4zxQbcFoE7uP8"], "labels": [], "masters": [], "uuids": ["a58bbf76-2856-4671-9910-99c9f398d408"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:15.241535", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136863 s, 843 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136863 s, 843 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:15.713 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IHO83mNjuenH3YCxAI5rzh16uTpPXouiR'], 'uuids': ['b5238c2d-384f-4e1b-b48d-08ba55d87ea0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.140099", "end": "2020-11-16 21:26:15.697076", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IHO83mNjuenH3YCxAI5rzh16uTpPXouiR"], "labels": [], "masters": [], "uuids": ["b5238c2d-384f-4e1b-b48d-08ba55d87ea0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:15.556977", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137539 s, 839 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137539 s, 839 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:15.725 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0', 'wwn-0x5000c50091e1af53'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QHP0', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e1af53', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1', 'wwn-0x5000c50091e1af53-part1'], 'uuids': ['9bdf28e7-5b1a-4776-884c-191bd8ad39f5'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0", "wwn-0x5000c50091e1af53"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1", "wwn-0x5000c50091e1af53-part1"], "labels": [], "masters": [], "uuids": ["9bdf28e7-5b1a-4776-884c-191bd8ad39f5"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHP0", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1af53"}}, 2020-11-16T21:26:15.727 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-11-16T21:26:16.037 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IFIHKt9jcRFn0Y2viegiNz3ExVXSERVeK'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.138052", "end": "2020-11-16 21:26:16.020481", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-svC6tnwv6jUPTbL4onmbHQFor8k4mZ4IFIHKt9jcRFn0Y2viegiNz3ExVXSERVeK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 21:26:15.882429", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135649 s, 850 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135649 s, 850 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:16.075 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:16 +0000 (0:00:01.905) 0:12:03.411 ******* 2020-11-16T21:26:20.372 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-16T21:26:20.374 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.064543", "end": "2020-11-16 21:26:20.351359", "rc": 0, "start": "2020-11-16 21:26:20.286816", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:20.403 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:20 +0000 (0:00:04.327) 0:12:07.739 ******* 2020-11-16T21:26:20.637 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-16T21:26:20.639 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043652", "end": "2020-11-16 21:26:20.622904", "rc": 0, "start": "2020-11-16 21:26:20.579252", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:20.668 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:20 +0000 (0:00:00.265) 0:12:08.004 ******* 2020-11-16T21:26:20.695 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-16T21:26:20.730 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:20 +0000 (0:00:00.061) 0:12:08.066 ******* 2020-11-16T21:26:20.757 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-16T21:26:20.759 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:20.791 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:20 +0000 (0:00:00.061) 0:12:08.127 ******* 2020-11-16T21:26:20.818 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-16T21:26:20.820 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:20.850 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:20 +0000 (0:00:00.058) 0:12:08.186 ******* 2020-11-16T21:26:20.877 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:20.911 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:20 +0000 (0:00:00.060) 0:12:08.247 ******* 2020-11-16T21:26:20.938 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-16T21:26:20.940 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:20.973 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:20 +0000 (0:00:00.061) 0:12:08.309 ******* 2020-11-16T21:26:21.165 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-16T21:26:21.167 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T21:26:21.202 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.228) 0:12:08.538 ******* 2020-11-16T21:26:21.249 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-16T21:26:21.251 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930529370112, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227180022, 'block_used': 12915086, 'inode_total': 61054976, 'inode_available': 60990124, 'inode_used': 64852, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227180022, "block_size": 4096, "block_total": 240095108, "block_used": 12915086, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990124, "inode_total": 61054976, "inode_used": 64852, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930529370112, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:21.284 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.082) 0:12:08.620 ******* 2020-11-16T21:26:21.311 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-16T21:26:21.313 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:21.345 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.060) 0:12:08.681 ******* 2020-11-16T21:26:21.371 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:26:21.373 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:21.403 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.058) 0:12:08.739 ******* 2020-11-16T21:26:21.429 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:26:21.431 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:21.462 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.059) 0:12:08.798 ******* 2020-11-16T21:26:21.489 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-16T21:26:21.490 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:21.521 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.058) 0:12:08.857 ******* 2020-11-16T21:26:21.547 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-16T21:26:21.549 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:21.579 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.058) 0:12:08.915 ******* 2020-11-16T21:26:21.605 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-16T21:26:21.607 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:21.638 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:21 +0000 (0:00:00.058) 0:12:08.974 ******* 2020-11-16T21:26:26.375 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-16T21:26:26.377 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-16T21:26:26.406 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:26 +0000 (0:00:04.768) 0:12:13.742 ******* 2020-11-16T21:26:27.031 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-16T21:26:27.033 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T21:26:31.445 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T21:26:39.924 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T21:26:40.244 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T21:26:40.566 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T21:26:40.599 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:40 +0000 (0:00:14.192) 0:12:27.935 ******* 2020-11-16T21:26:40.791 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-16T21:26:40.792 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-16T21:26:40.822 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:40 +0000 (0:00:00.223) 0:12:28.158 ******* 2020-11-16T21:26:41.026 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-16T21:26:41.028 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T21:26:41.214 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T21:26:41.390 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T21:26:41.566 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T21:26:41.577 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:41.611 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:41 +0000 (0:00:00.788) 0:12:28.947 ******* 2020-11-16T21:26:41.813 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-16T21:26:41.815 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-16T21:26:41.844 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:41 +0000 (0:00:00.233) 0:12:29.180 ******* 2020-11-16T21:26:41.881 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-16T21:26:41.883 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-16T21:26:41.914 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:41 +0000 (0:00:00.070) 0:12:29.251 ******* 2020-11-16T21:26:43.555 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-16T21:26:43.557 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true} 2020-11-16T21:26:43.588 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:43 +0000 (0:00:01.673) 0:12:30.924 ******* 2020-11-16T21:26:44.150 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-16T21:26:44.152 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-16T21:26:44.182 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.593) 0:12:31.518 ******* 2020-11-16T21:26:44.209 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-16T21:26:44.211 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.240 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.058) 0:12:31.576 ******* 2020-11-16T21:26:44.267 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-16T21:26:44.269 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.298 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.057) 0:12:31.634 ******* 2020-11-16T21:26:44.324 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-16T21:26:44.326 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.356 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.058) 0:12:31.692 ******* 2020-11-16T21:26:44.383 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-16T21:26:44.384 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.415 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.058) 0:12:31.751 ******* 2020-11-16T21:26:44.442 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-16T21:26:44.443 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.473 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.058) 0:12:31.809 ******* 2020-11-16T21:26:44.499 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-16T21:26:44.501 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.530 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.056) 0:12:31.866 ******* 2020-11-16T21:26:44.556 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-16T21:26:44.558 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.587 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.057) 0:12:31.923 ******* 2020-11-16T21:26:44.617 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-16T21:26:44.619 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.648 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.060) 0:12:31.984 ******* 2020-11-16T21:26:44.675 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-16T21:26:44.677 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.706 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.058) 0:12:32.042 ******* 2020-11-16T21:26:44.733 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-16T21:26:44.735 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.765 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.058) 0:12:32.100 ******* 2020-11-16T21:26:44.791 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-16T21:26:44.793 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:44.823 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:44 +0000 (0:00:00.058) 0:12:32.159 ******* 2020-11-16T21:26:45.283 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-16T21:26:45.285 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-11-16T21:26:45.315 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:45 +0000 (0:00:00.491) 0:12:32.651 ******* 2020-11-16T21:26:45.622 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-16T21:26:45.623 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-16 20:21:24 UTC", "ActiveEnterTimestampMonotonic": "14963156", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount sntp.service tmp.mount systemd-journald.socket system.slice sysinit.target basic.target systemd-tmpfiles-setup.service ntpd.service ntpdate.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-16 20:21:23 UTC", "AssertTimestampMonotonic": "14098604", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-16 20:21:23 UTC", "ConditionTimestampMonotonic": "14098583", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-11-16T21:26:45.624 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2457", "ExecMainStartTimestamp": "Mon 2020-11-16 20:21:24 UTC", "ExecMainStartTimestampMonotonic": "14907687", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-16 20:21:23 UTC", "InactiveExitTimestampMonotonic": "14160760", "InvocationID": "ab53d6c96f4a44cd864125c473aec613", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-11-16T21:26:45.626 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127425", "LimitNPROCSoft": "127425", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127425", "LimitSIGPENDINGSoft": "127425", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2457", "MemoryAccounting": "yes", "MemoryCurrent": "2584576", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-11-16T21:26:45.627 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-16 20:21:24 UTC", "StateChangeTimestampMonotonic": "14963156", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-16 20:21:24 UTC", "WatchdogTimestampMonotonic": "14907701", "WatchdogUSec": "0"}} 2020-11-16T21:26:45.655 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:45 +0000 (0:00:00.340) 0:12:32.991 ******* 2020-11-16T21:26:45.682 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:26:45.684 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-16T21:26:45.713 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:45 +0000 (0:00:00.057) 0:12:33.049 ******* 2020-11-16T21:26:45.742 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:26:45.743 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-11-16T21:26:45.772 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:45 +0000 (0:00:00.059) 0:12:33.108 ******* 2020-11-16T21:26:45.799 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-16T21:26:45.801 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:45.831 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:45 +0000 (0:00:00.058) 0:12:33.167 ******* 2020-11-16T21:26:45.857 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-16T21:26:45.858 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:45.888 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:45 +0000 (0:00:00.057) 0:12:33.224 ******* 2020-11-16T21:26:45.914 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-16T21:26:45.916 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:45.945 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:45 +0000 (0:00:00.057) 0:12:33.281 ******* 2020-11-16T21:26:46.147 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-16T21:26:46.151 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-11-16T21:26:46.182 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:46 +0000 (0:00:00.236) 0:12:33.518 ******* 2020-11-16T21:26:46.654 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-16T21:26:46.655 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-11-16T21:26:46.685 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:46 +0000 (0:00:00.503) 0:12:34.021 ******* 2020-11-16T21:26:52.013 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-16T21:26:52.015 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-16T21:26:52.044 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:05.359) 0:12:39.380 ******* 2020-11-16T21:26:52.071 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-16T21:26:52.072 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:52.102 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.057) 0:12:39.438 ******* 2020-11-16T21:26:52.129 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-16T21:26:52.131 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-16T21:26:52.160 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.058) 0:12:39.496 ******* 2020-11-16T21:26:52.353 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-16T21:26:52.355 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-11-16T21:26:52.385 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.224) 0:12:39.721 ******* 2020-11-16T21:26:52.411 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-16T21:26:52.413 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:52.442 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.057) 0:12:39.778 ******* 2020-11-16T21:26:52.469 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-16T21:26:52.471 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-16T21:26:52.499 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.057) 0:12:39.835 ******* 2020-11-16T21:26:52.525 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-16T21:26:52.527 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:52.556 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.056) 0:12:39.892 ******* 2020-11-16T21:26:52.749 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-16T21:26:52.751 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-16T21:26:52.780 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.224) 0:12:40.116 ******* 2020-11-16T21:26:52.973 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-16T21:26:52.974 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-16T21:26:53.005 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:52 +0000 (0:00:00.224) 0:12:40.341 ******* 2020-11-16T21:26:53.031 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-16T21:26:53.033 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.062 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.057) 0:12:40.398 ******* 2020-11-16T21:26:53.091 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-16T21:26:53.093 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.102 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.104 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.113 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.145 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.082) 0:12:40.481 ******* 2020-11-16T21:26:53.182 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-16T21:26:53.184 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.194 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.196 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.227 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.081) 0:12:40.563 ******* 2020-11-16T21:26:53.254 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T21:26:53.255 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.287 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.059) 0:12:40.622 ******* 2020-11-16T21:26:53.313 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-16T21:26:53.315 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.344 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.057) 0:12:40.680 ******* 2020-11-16T21:26:53.371 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-16T21:26:53.373 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.404 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.059) 0:12:40.740 ******* 2020-11-16T21:26:53.431 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.462 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.057) 0:12:40.798 ******* 2020-11-16T21:26:53.488 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-16T21:26:53.490 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.521 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.059) 0:12:40.857 ******* 2020-11-16T21:26:53.547 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-16T21:26:53.548 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.580 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.059) 0:12:40.916 ******* 2020-11-16T21:26:53.606 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-16T21:26:53.608 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:53.642 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:53 +0000 (0:00:00.061) 0:12:40.978 ******* 2020-11-16T21:26:57.696 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-16T21:26:57.698 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T21:26:57.727 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:57 +0000 (0:00:04.085) 0:12:45.063 ******* 2020-11-16T21:26:58.957 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-16T21:26:58.959 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006017", "end": "2020-11-16 21:26:58.925579", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-16 21:26:57.919562", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:59.143 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002991", "end": "2020-11-16 21:26:59.119093", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-16 21:26:59.116102", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:59.319 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002721", "end": "2020-11-16 21:26:59.294033", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-16 21:26:59.291312", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:59.515 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026789", "end": "2020-11-16 21:26:59.493364", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-16 21:26:59.466575", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T21:26:59.549 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:59 +0000 (0:00:01.821) 0:12:46.885 ******* 2020-11-16T21:26:59.575 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-16T21:26:59.606 INFO:teuthology.task.ansible.out:Monday 16 November 2020 21:26:59 +0000 (0:00:00.057) 0:12:46.942 ******* 2020-11-16T21:26:59.798 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-16T21:26:59.800 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-16T21:26:59.886 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-16T21:26:59.889 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-16T21:26:59.893 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-11-16T21:26:59.895 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi120.front.sepia.ceph.com : ok=111 changed=31 unreachable=0 failed=0 skipped=178 rescued=0 ignored=3 Monday 16 November 2020 21:26:59 +0000 (0:00:00.287) 0:12:47.230 ******* =============================================================================== Gathering Facts ------------------------------------------------------- 134.72s common : nrpe - Copy SELinux type enforcement file --------------------- 75.41s common : Install Kerberos Packages (RedHat) ---------------------------- 38.79s users : Determine if we can skip users and pubkeys updates ------------- 33.51s common : Configure local epel mirrorlists ------------------------------ 21.78s testnode : Configure /etc/sudoers. ------------------------------------- 20.74s testnode : Upload ssh_config ------------------------------------------- 20.66s common : nrpe - Load SELinux policy package ---------------------------- 18.99s ansible-managed : Create the cephlab_sudo sudoers.d file. -------------- 17.81s common : Set the hardware clock ---------------------------------------- 17.40s testnode : Ensure the group kvm exists. -------------------------------- 16.41s common : nrpe - Remove SELinux policy package -------------------------- 16.39s testnode : Create logical volume(s) ------------------------------------ 14.19s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. -- 14.05s testnode : Configure local mirrorlists --------------------------------- 13.56s testnode : Configure version specific repos in /etc/yum.repos.d/ ------- 13.45s users : Write /keys-repo-sha1 sentinel file ---------------------------- 13.36s users : Create all users without sudo access. -------------------------- 12.40s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. -- 10.03s common : nrpe - Set SELinux b 2020-11-16T21:26:59.896 INFO:teuthology.task.ansible.out:oolean nagios_run_sudo true --------------- 10.03s 2020-11-16T21:27:00.209 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2020-11-16T21:27:00.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T21:27:00.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T21:27:00.285 INFO:teuthology.orchestra.run.smithi120:> true 2020-11-16T21:27:00.371 INFO:teuthology.run_tasks:Running task clock... 2020-11-16T21:27:00.387 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-16T21:27:00.387 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-16T21:27:00.388 INFO:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-16T21:27:00.444 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-11-16T21:27:00.462 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-11-16T21:27:00.502 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2020-11-16T21:27:00.516 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2020-11-16T21:27:00.534 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-11-16T21:27:00.551 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-11-16T21:27:00.615 INFO:teuthology.orchestra.run.smithi120.stderr:bash: ntpq: command not found 2020-11-16T21:27:00.618 INFO:teuthology.orchestra.run.smithi120.stdout:210 Number of sources = 4 2020-11-16T21:27:00.618 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-11-16T21:27:00.618 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2020-11-16T21:27:00.618 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-16T21:27:00.618 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-16T21:27:00.618 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-16T21:27:00.619 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-16T21:27:00.619 INFO:teuthology.run_tasks:Running task install... 2020-11-16T21:27:00.627 DEBUG:teuthology.task.install:project ceph 2020-11-16T21:27:00.628 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '911cc35794d871071a8bebfb0919482263b1d18b'}} 2020-11-16T21:27:00.628 DEBUG:teuthology.task.install:config {'sha1': '911cc35794d871071a8bebfb0919482263b1d18b'} 2020-11-16T21:27:00.628 INFO:teuthology.task.install:Using flavor: basic 2020-11-16T21:27:00.634 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-16T21:27:00.635 INFO:teuthology.task.install:extra packages: [] 2020-11-16T21:27:00.635 INFO:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2020-11-16T21:27:00.696 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS 2020-11-16T21:27:00.697 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-16T21:27:00.697 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '911cc35794d871071a8bebfb0919482263b1d18b', 'tag': None, 'wait_for_package': False} 2020-11-16T21:27:00.698 INFO:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2020-11-16T21:27:00.772 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS 2020-11-16T21:27:00.773 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-16T21:27:00.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T21:27:00.943 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/centos/8/flavors/default/ 2020-11-16T21:27:00.944 INFO:teuthology.task.install.rpm:Package version is 16.0.0-7219.g911cc357 2020-11-16T21:27:01.069 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-11-16T21:27:01.070 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:27:01.071 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-16T21:27:01.084 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-16T21:27:01.097 INFO:teuthology.orchestra.run.smithi120.stderr:1+1 records in 2020-11-16T21:27:01.097 INFO:teuthology.orchestra.run.smithi120.stderr:1+1 records out 2020-11-16T21:27:01.097 INFO:teuthology.orchestra.run.smithi120.stderr:664 bytes copied, 3.5542e-05 s, 18.7 MB/s 2020-11-16T21:27:01.098 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-11-16T21:27:01.098 INFO:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/911cc35794d871071a8bebfb0919482263b1d18b/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-11-16T21:27:01.165 INFO:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-11-16T21:27:01.246 INFO:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-11-16T21:27:01.313 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2020-11-16T21:27:01.315 INFO:teuthology.orchestra.run.smithi120:> sudo yum clean all 2020-11-16T21:27:01.592 INFO:teuthology.orchestra.run.smithi120.stdout:42 files removed 2020-11-16T21:27:01.620 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw 2020-11-16T21:27:02.209 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 293 kB/s | 72 kB 00:00 2020-11-16T21:27:02.442 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 72 kB/s | 15 kB 00:00 2020-11-16T21:27:02.669 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 7.0 kB/s | 1.4 kB 00:00 2020-11-16T21:27:02.830 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - AppStream 49 MB/s | 7.0 MB 00:00 2020-11-16T21:27:04.026 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Base 25 MB/s | 2.2 MB 00:00 2020-11-16T21:27:04.457 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Extras 87 kB/s | 5.5 kB 00:00 2020-11-16T21:27:04.877 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 22 MB/s | 8.3 MB 00:00 2020-11-16T21:27:06.628 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 4.3 MB/s | 23 kB 00:00 2020-11-16T21:27:07.643 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:27:07.660 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:07.660 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-11-16T21:27:07.660 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:07.661 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:27:07.661 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:16.0.0-7219.g911cc357.el8 ceph 12 M 2020-11-16T21:27:07.661 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2020-11-16T21:27:07.661 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:16.0.0-7219.g911cc357.el8 ceph 3.7 M 2020-11-16T21:27:07.661 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:16.0.0-7219.g911cc357.el8 ceph 2.4 M 2020-11-16T21:27:07.661 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:27:07.661 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:16.0.0-7219.g911cc357.el8 ceph 5.9 M 2020-11-16T21:27:07.662 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:16.0.0-7219.g911cc357.el8 ceph 23 M 2020-11-16T21:27:07.662 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:16.0.0-7219.g911cc357.el8 ceph 25 k 2020-11-16T21:27:07.662 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:16.0.0-7219.g911cc357.el8 ceph 554 k 2020-11-16T21:27:07.662 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:16.0.0-7219.g911cc357.el8 ceph 480 k 2020-11-16T21:27:07.662 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:16.0.0-7219.g911cc357.el8 ceph 3.7 M 2020-11-16T21:27:07.662 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse 2020-11-16T21:27:07.662 INFO:teuthology.orchestra.run.smithi120.stdout: x86_64 2:16.0.0-7219.g911cc357.el8 ceph 40 k 2020-11-16T21:27:07.663 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:16.0.0-7219.g911cc357.el8 ceph 73 k 2020-11-16T21:27:07.663 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:16.0.0-7219.g911cc357.el8 ceph 204 k 2020-11-16T21:27:07.663 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:16.0.0-7219.g911cc357.el8 ceph 375 k 2020-11-16T21:27:07.663 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:16.0.0-7219.g911cc357.el8 ceph 353 k 2020-11-16T21:27:07.663 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:16.0.0-7219.g911cc357.el8 ceph 113 k 2020-11-16T21:27:07.663 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-11-16T21:27:07.663 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-11-16T21:27:07.664 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-11-16T21:27:07.664 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-11-16T21:27:07.664 INFO:teuthology.orchestra.run.smithi120.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2020-11-16T21:27:07.664 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-11-16T21:27:07.664 INFO:teuthology.orchestra.run.smithi120.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2020-11-16T21:27:07.664 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-11-16T21:27:07.664 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-11-16T21:27:07.665 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:07.665 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:27:07.665 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:07.665 INFO:teuthology.orchestra.run.smithi120.stdout:Install 22 Packages 2020-11-16T21:27:07.665 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 2 Packages 2020-11-16T21:27:07.665 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:07.666 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 54 M 2020-11-16T21:27:07.666 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:27:08.373 INFO:teuthology.orchestra.run.smithi120.stdout:(1/24): ceph-base-16.0.0-7219.g911cc357.el8.x86 8.3 MB/s | 5.9 MB 00:00 2020-11-16T21:27:08.410 INFO:teuthology.orchestra.run.smithi120.stdout:(2/24): ceph-selinux-16.0.0-7219.g911cc357.el8. 662 kB/s | 25 kB 00:00 2020-11-16T21:27:08.468 INFO:teuthology.orchestra.run.smithi120.stdout:(3/24): libcephfs2-16.0.0-7219.g911cc357.el8.x8 9.5 MB/s | 554 kB 00:00 2020-11-16T21:27:08.523 INFO:teuthology.orchestra.run.smithi120.stdout:(4/24): libradosstriper1-16.0.0-7219.g911cc357. 8.5 MB/s | 480 kB 00:00 2020-11-16T21:27:08.847 INFO:teuthology.orchestra.run.smithi120.stdout:(5/24): librgw2-16.0.0-7219.g911cc357.el8.x86_6 11 MB/s | 3.7 MB 00:00 2020-11-16T21:27:08.889 INFO:teuthology.orchestra.run.smithi120.stdout:(6/24): python3-ceph-argparse-16.0.0-7219.g911c 956 kB/s | 40 kB 00:00 2020-11-16T21:27:08.930 INFO:teuthology.orchestra.run.smithi120.stdout:(7/24): python3-ceph-common-16.0.0-7219.g911cc3 1.8 MB/s | 73 kB 00:00 2020-11-16T21:27:08.976 INFO:teuthology.orchestra.run.smithi120.stdout:(8/24): python3-cephfs-16.0.0-7219.g911cc357.el 4.3 MB/s | 204 kB 00:00 2020-11-16T21:27:09.030 INFO:teuthology.orchestra.run.smithi120.stdout:(9/24): python3-rados-16.0.0-7219.g911cc357.el8 6.8 MB/s | 375 kB 00:00 2020-11-16T21:27:09.083 INFO:teuthology.orchestra.run.smithi120.stdout:(10/24): python3-rbd-16.0.0-7219.g911cc357.el8. 6.5 MB/s | 353 kB 00:00 2020-11-16T21:27:09.128 INFO:teuthology.orchestra.run.smithi120.stdout:(11/24): python3-rgw-16.0.0-7219.g911cc357.el8. 2.5 MB/s | 113 kB 00:00 2020-11-16T21:27:09.161 INFO:teuthology.orchestra.run.smithi120.stdout:(12/24): librdkafka-0.11.4-1.el8.x86_64.rpm 11 MB/s | 353 kB 00:00 2020-11-16T21:27:09.165 INFO:teuthology.orchestra.run.smithi120.stdout:(13/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 67 MB/s | 259 kB 00:00 2020-11-16T21:27:09.166 INFO:teuthology.orchestra.run.smithi120.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 33 MB/s | 44 kB 00:00 2020-11-16T21:27:09.167 INFO:teuthology.orchestra.run.smithi120.stdout:(15/24): librabbitmq-0.9.0-1.el8.x86_64.rpm 36 MB/s | 47 kB 00:00 2020-11-16T21:27:09.248 INFO:teuthology.orchestra.run.smithi120.stdout:(16/24): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2020-11-16T21:27:09.292 INFO:teuthology.orchestra.run.smithi120.stdout:(17/24): ceph-radosgw-16.0.0-7219.g911cc357.el8 7.2 MB/s | 12 MB 00:01 2020-11-16T21:27:09.297 INFO:teuthology.orchestra.run.smithi120.stdout:(18/24): gperftools-libs-2.7-6.el8.x86_64.rpm 6.3 MB/s | 315 kB 00:00 2020-11-16T21:27:09.314 INFO:teuthology.orchestra.run.smithi120.stdout:(19/24): liboath-2.6.2-3.el8.x86_64.rpm 3.4 MB/s | 59 kB 00:00 2020-11-16T21:27:09.331 INFO:teuthology.orchestra.run.smithi120.stdout:(20/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.3 MB/s | 75 kB 00:00 2020-11-16T21:27:09.373 INFO:teuthology.orchestra.run.smithi120.stdout:(21/24): leveldb-1.22-1.el8.x86_64.rpm 2.2 MB/s | 181 kB 00:00 2020-11-16T21:27:09.681 INFO:teuthology.orchestra.run.smithi120.stdout:(22/24): ceph-common-16.0.0-7219.g911cc357.el8. 11 MB/s | 23 MB 00:02 2020-11-16T21:27:09.722 INFO:teuthology.orchestra.run.smithi120.stdout:(23/24): librados2-16.0.0-7219.g911cc357.el8.x8 9.4 MB/s | 3.7 MB 00:00 2020-11-16T21:27:09.729 INFO:teuthology.orchestra.run.smithi120.stdout:(24/24): librbd1-16.0.0-7219.g911cc357.el8.x86_ 6.7 MB/s | 2.4 MB 00:00 2020-11-16T21:27:09.732 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:27:09.732 INFO:teuthology.orchestra.run.smithi120.stdout:Total 26 MB/s | 54 MB 00:02 2020-11-16T21:27:09.738 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:27:09.782 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:27:09.782 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:27:10.201 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:27:10.201 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:27:11.219 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:27:11.372 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2020-11-16T21:27:11.559 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2020-11-16T21:27:11.725 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-11-16T21:27:12.084 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-11-16T21:27:12.275 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:16.0.0-7219.g911cc357.el8.x86_64 4/26 2020-11-16T21:27:12.371 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.0.0-7219.g911cc357.el8.x86_64 4/26 2020-11-16T21:27:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:16.0.0-7219.g911cc357.el8.x86_64 5/26 2020-11-16T21:27:12.708 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:16.0.0-7219.g911cc357.el8.x86_64 6/26 2020-11-16T21:27:12.977 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:16.0.0-7219.g911cc357.el8.x86_64 6/26 2020-11-16T21:27:13.167 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:16.0.0-7219.g911cc357.el8.x86_64 7/26 2020-11-16T21:27:13.236 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.0.0-7219.g911cc357.el8.x86_64 7/26 2020-11-16T21:27:13.433 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 8/26 2020-11-16T21:27:13.575 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2020-11-16T21:27:13.936 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2020-11-16T21:27:14.075 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:16.0.0-7219.g911cc357.el8.x86_64 10/26 2020-11-16T21:27:14.143 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:16.0.0-7219.g911cc357.el8.x86_64 10/26 2020-11-16T21:27:14.296 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2020-11-16T21:27:14.429 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:16.0.0-7219.g911cc357.el8. 12/26 2020-11-16T21:27:14.595 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:16.0.0-7219.g911cc357.el8.x86_64 13/26 2020-11-16T21:27:15.008 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:16.0.0-7219.g911cc357.el8.x86_64 14/26 2020-11-16T21:27:15.207 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:16.0.0-7219.g911cc357.el8.x86_64 15/26 2020-11-16T21:27:15.350 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:16.0.0-7219.g911cc357.el8.x86_6 16/26 2020-11-16T21:27:15.404 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:16.0.0-7219.g911cc357.el8.x86_6 16/26 2020-11-16T21:27:15.562 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2020-11-16T21:27:15.716 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 18/26 2020-11-16T21:27:15.897 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2020-11-16T21:27:16.033 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:16.0.0-7219.g911cc357.el8.x8 20/26 2020-11-16T21:27:19.710 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:16.0.0-7219.g911cc357.el8.x86_64 21/26 2020-11-16T21:27:19.908 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:16.0.0-7219.g911cc357.el8.x86_64 21/26 2020-11-16T21:27:20.569 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:16.0.0-7219.g911cc357.el8.x86_64 21/26 2020-11-16T21:27:20.840 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:16.0.0-7219.g911cc357.el8.x86_64 22/26 2020-11-16T21:27:20.911 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:16.0.0-7219.g911cc357.el8.x86_64 22/26 2020-11-16T21:27:21.041 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:16.0.0-7219.g911cc357.el8.x86_64 23/26 2020-11-16T21:27:37.762 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:16.0.0-7219.g911cc357.el8.x86_64 23/26 2020-11-16T21:27:37.762 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2020-11-16T21:27:37.762 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2020-11-16T21:27:37.762 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /dev 2020-11-16T21:27:37.763 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /run 2020-11-16T21:27:37.763 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2020-11-16T21:27:37.763 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2020-11-16T21:27:37.763 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2020-11-16T21:27:37.763 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2020-11-16T21:27:37.763 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /dev 2020-11-16T21:27:37.763 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:38.905 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:16.0.0-7219.g911cc357.el8.x86_64 24/26 2020-11-16T21:27:38.963 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-7219.g911cc357.el8.x86_64 24/26 2020-11-16T21:27:38.963 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-11-16T21:27:39.140 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-11-16T21:27:39.140 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2020-11-16T21:27:39.819 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2020-11-16T21:27:39.820 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:16.0.0-7219.g911cc357.el8.x86_64 1/26 2020-11-16T21:27:39.820 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:16.0.0-7219.g911cc357.el8.x86_64 2/26 2020-11-16T21:27:39.820 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:16.0.0-7219.g911cc357.el8.x86_64 3/26 2020-11-16T21:27:39.820 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:16.0.0-7219.g911cc357.el8.x86_64 4/26 2020-11-16T21:27:39.820 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:16.0.0-7219.g911cc357.el8.x86_64 5/26 2020-11-16T21:27:39.820 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:16.0.0-7219.g911cc357.el8.x86_6 6/26 2020-11-16T21:27:39.821 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:16.0.0-7219.g911cc357.el8.x86_64 7/26 2020-11-16T21:27:39.821 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:16.0.0-7219.g911cc357.el8. 8/26 2020-11-16T21:27:39.821 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:16.0.0-7219.g911cc357.el8.x8 9/26 2020-11-16T21:27:39.821 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:16.0.0-7219.g911cc357.el8.x86_64 10/26 2020-11-16T21:27:39.821 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:16.0.0-7219.g911cc357.el8.x86_64 11/26 2020-11-16T21:27:39.821 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:16.0.0-7219.g911cc357.el8.x86_64 12/26 2020-11-16T21:27:39.821 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:16.0.0-7219.g911cc357.el8.x86_64 13/26 2020-11-16T21:27:39.822 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2020-11-16T21:27:39.822 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2020-11-16T21:27:39.822 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2020-11-16T21:27:39.822 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/26 2020-11-16T21:27:39.822 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2020-11-16T21:27:39.822 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 19/26 2020-11-16T21:27:39.822 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2020-11-16T21:27:39.823 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2020-11-16T21:27:39.823 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2020-11-16T21:27:39.823 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.0.0-7219.g911cc357.el8.x86_64 23/26 2020-11-16T21:27:39.823 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2020-11-16T21:27:39.823 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.0.0-7219.g911cc357.el8.x86_64 25/26 2020-11-16T21:27:40.710 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2020-11-16T21:27:40.711 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:40.711 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2020-11-16T21:27:40.711 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.711 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.711 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:40.711 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:27:40.711 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.712 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.712 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.712 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.712 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.712 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.712 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.712 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.713 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.713 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.713 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.713 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.713 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:27:40.713 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-11-16T21:27:40.713 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-11-16T21:27:40.714 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-11-16T21:27:40.714 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-11-16T21:27:40.714 INFO:teuthology.orchestra.run.smithi120.stdout: fmt-6.2.1-1.el8.x86_64 2020-11-16T21:27:40.714 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-11-16T21:27:40.714 INFO:teuthology.orchestra.run.smithi120.stdout: leveldb-1.22-1.el8.x86_64 2020-11-16T21:27:40.714 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.2-3.el8.x86_64 2020-11-16T21:27:40.714 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.3.1-3.el8.x86_64 2020-11-16T21:27:40.715 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:40.715 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:27:41.034 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-test 2020-11-16T21:27:41.458 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:35 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:27:41.825 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:27:41.826 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:41.826 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-11-16T21:27:41.827 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:41.827 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:27:41.827 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:16.0.0-7219.g911cc357.el8 ceph 47 M 2020-11-16T21:27:41.827 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:27:41.827 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-11-16T21:27:41.827 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-11-16T21:27:41.827 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-11-16T21:27:41.828 INFO:teuthology.orchestra.run.smithi120.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-11-16T21:27:41.828 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-11-16T21:27:41.828 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:41.828 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:27:41.828 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:41.828 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2020-11-16T21:27:41.828 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:41.829 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 48 M 2020-11-16T21:27:41.829 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 173 M 2020-11-16T21:27:41.829 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:27:41.889 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 2.7 MB/s | 161 kB 00:00 2020-11-16T21:27:41.893 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 2.9 MB/s | 188 kB 00:00 2020-11-16T21:27:41.896 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 39 MB/s | 298 kB 00:00 2020-11-16T21:27:41.898 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 53 MB/s | 249 kB 00:00 2020-11-16T21:27:41.963 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.0 MB/s | 69 kB 00:00 2020-11-16T21:27:44.086 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-test-16.0.0-7219.g911cc357.el8.x86_ 21 MB/s | 47 MB 00:02 2020-11-16T21:27:44.089 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:27:44.089 INFO:teuthology.orchestra.run.smithi120.stdout:Total 21 MB/s | 48 MB 00:02 2020-11-16T21:27:44.095 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:27:44.117 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:27:44.118 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:27:44.415 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:27:44.416 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:27:44.861 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:27:45.033 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-11-16T21:27:45.222 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-11-16T21:27:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-11-16T21:27:45.504 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-11-16T21:27:45.576 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-11-16T21:27:50.064 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-11-16T21:27:50.283 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:16.0.0-7219.g911cc357.el8.x86_64 6/6 2020-11-16T21:27:50.476 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-test-2:16.0.0-7219.g911cc357.el8.x86_64 6/6 2020-11-16T21:27:50.476 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:16.0.0-7219.g911cc357.el8.x86_64 1/6 2020-11-16T21:27:50.476 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-11-16T21:27:50.477 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-11-16T21:27:50.477 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-11-16T21:27:50.477 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-11-16T21:27:50.799 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-11-16T21:27:50.799 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:50.799 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:27:50.799 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:16.0.0-7219.g911cc357.el8.x86_64 jq-1.5-12.el8.x86_64 2020-11-16T21:27:50.800 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-11-16T21:27:50.800 INFO:teuthology.orchestra.run.smithi120.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-11-16T21:27:50.800 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:50.800 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:27:51.020 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph 2020-11-16T21:27:51.431 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:45 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:27:51.864 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:27:51.868 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:51.868 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-11-16T21:27:51.868 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:51.868 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:27:51.869 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:16.0.0-7219.g911cc357.el8 ceph 6.4 k 2020-11-16T21:27:51.869 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:27:51.869 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:16.0.0-7219.g911cc357.el8 ceph 2.1 M 2020-11-16T21:27:51.869 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:16.0.0-7219.g911cc357.el8 ceph 1.5 M 2020-11-16T21:27:51.869 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:16.0.0-7219.g911cc357.el8 ceph 4.9 M 2020-11-16T21:27:51.869 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:16.0.0-7219.g911cc357.el8 ceph 18 M 2020-11-16T21:27:51.869 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core 2020-11-16T21:27:51.870 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 2:16.0.0-7219.g911cc357.el8 ceph-noarch 198 k 2020-11-16T21:27:51.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2020-11-16T21:27:51.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2020-11-16T21:27:51.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2020-11-16T21:27:51.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2020-11-16T21:27:51.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.24.2-2.el8 CentOS-Base 176 k 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-beautifulsoup4 2020-11-16T21:27:51.871 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 4.6.3-2.el8.1 epel 185 k 2020-11-16T21:27:51.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.2.1-1.el8 epel 145 k 2020-11-16T21:27:51.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2020-11-16T21:27:51.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2020-11-16T21:27:51.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools 2020-11-16T21:27:51.872 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 2.0-4.el8 epel 18 k 2020-11-16T21:27:51.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2020-11-16T21:27:51.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools 2020-11-16T21:27:51.873 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 7.2.0-3.el8 epel 59 k 2020-11-16T21:27:51.873 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2020-11-16T21:27:51.873 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2020-11-16T21:27:51.873 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplegeneric 2020-11-16T21:27:51.873 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 0.8.1-17.el8 epel 19 k 2020-11-16T21:27:51.873 INFO:teuthology.orchestra.run.smithi120.stdout: python3-singledispatch 2020-11-16T21:27:51.873 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 3.4.0.3-18.el8 epel 24 k 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2020-11-16T21:27:51.874 INFO:teuthology.orchestra.run.smithi120.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-11-16T21:27:51.875 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2020-11-16T21:27:51.875 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:51.875 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:27:51.875 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:27:51.875 INFO:teuthology.orchestra.run.smithi120.stdout:Install 36 Packages 2020-11-16T21:27:51.875 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:27:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 31 M 2020-11-16T21:27:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 101 M 2020-11-16T21:27:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:27:51.975 INFO:teuthology.orchestra.run.smithi120.stdout:(1/36): ceph-16.0.0-7219.g911cc357.el8.x86_64.r 64 kB/s | 6.4 kB 00:00 2020-11-16T21:27:52.250 INFO:teuthology.orchestra.run.smithi120.stdout:(2/36): ceph-mds-16.0.0-7219.g911cc357.el8.x86_ 5.7 MB/s | 2.1 MB 00:00 2020-11-16T21:27:52.332 INFO:teuthology.orchestra.run.smithi120.stdout:(3/36): ceph-mgr-16.0.0-7219.g911cc357.el8.x86_ 3.3 MB/s | 1.5 MB 00:00 2020-11-16T21:27:52.444 INFO:teuthology.orchestra.run.smithi120.stdout:(4/36): ceph-mgr-modules-core-16.0.0-7219.g911c 1.7 MB/s | 198 kB 00:00 2020-11-16T21:27:52.532 INFO:teuthology.orchestra.run.smithi120.stdout:(5/36): ceph-mon-16.0.0-7219.g911cc357.el8.x86_ 8.9 MB/s | 4.9 MB 00:00 2020-11-16T21:27:52.561 INFO:teuthology.orchestra.run.smithi120.stdout:(6/36): python3-html5lib-0.999999999-6.el8.noar 1.8 MB/s | 214 kB 00:00 2020-11-16T21:27:52.636 INFO:teuthology.orchestra.run.smithi120.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 2.1 MB/s | 157 kB 00:00 2020-11-16T21:27:52.666 INFO:teuthology.orchestra.run.smithi120.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 1.3 MB/s | 39 kB 00:00 2020-11-16T21:27:52.760 INFO:teuthology.orchestra.run.smithi120.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 574 kB/s | 54 kB 00:00 2020-11-16T21:27:52.778 INFO:teuthology.orchestra.run.smithi120.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 1.5 MB/s | 27 kB 00:00 2020-11-16T21:27:52.826 INFO:teuthology.orchestra.run.smithi120.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 9.4 MB/s | 457 kB 00:00 2020-11-16T21:27:52.833 INFO:teuthology.orchestra.run.smithi120.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 30 MB/s | 195 kB 00:00 2020-11-16T21:27:52.835 INFO:teuthology.orchestra.run.smithi120.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 14 MB/s | 34 kB 00:00 2020-11-16T21:27:52.842 INFO:teuthology.orchestra.run.smithi120.stdout:(14/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 4.9 MB/s | 1.5 MB 00:00 2020-11-16T21:27:52.874 INFO:teuthology.orchestra.run.smithi120.stdout:(15/36): python3-requests-2.20.0-2.1.el8_1.noar 3.2 MB/s | 123 kB 00:00 2020-11-16T21:27:52.915 INFO:teuthology.orchestra.run.smithi120.stdout:(16/36): python3-urllib3-1.24.2-2.el8.noarch.rp 2.4 MB/s | 176 kB 00:00 2020-11-16T21:27:52.973 INFO:teuthology.orchestra.run.smithi120.stdout:(17/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 443 kB/s | 44 kB 00:00 2020-11-16T21:27:53.012 INFO:teuthology.orchestra.run.smithi120.stdout:(18/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-11-16T21:27:53.031 INFO:teuthology.orchestra.run.smithi120.stdout:(19/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.4 MB/s | 145 kB 00:00 2020-11-16T21:27:53.051 INFO:teuthology.orchestra.run.smithi120.stdout:(20/36): python3-cssselect-0.9.2-10.el8.noarch. 2.0 MB/s | 40 kB 00:00 2020-11-16T21:27:53.053 INFO:teuthology.orchestra.run.smithi120.stdout:(21/36): python3-cherrypy-18.4.0-1.el8.noarch.r 9.2 MB/s | 384 kB 00:00 2020-11-16T21:27:53.071 INFO:teuthology.orchestra.run.smithi120.stdout:(22/36): python3-jaraco-6.2-6.el8.noarch.rpm 569 kB/s | 11 kB 00:00 2020-11-16T21:27:53.073 INFO:teuthology.orchestra.run.smithi120.stdout:(23/36): python3-jaraco-functools-2.0-4.el8.noa 899 kB/s | 18 kB 00:00 2020-11-16T21:27:53.091 INFO:teuthology.orchestra.run.smithi120.stdout:(24/36): python3-logutils-0.3.5-11.el8.noarch.r 2.4 MB/s | 49 kB 00:00 2020-11-16T21:27:53.093 INFO:teuthology.orchestra.run.smithi120.stdout:(25/36): python3-more-itertools-7.2.0-3.el8.noa 2.9 MB/s | 59 kB 00:00 2020-11-16T21:27:53.113 INFO:teuthology.orchestra.run.smithi120.stdout:(26/36): python3-portend-2.6-1.el8.noarch.rpm 797 kB/s | 16 kB 00:00 2020-11-16T21:27:53.131 INFO:teuthology.orchestra.run.smithi120.stdout:(27/36): python3-pecan-1.3.2-9.el8.noarch.rpm 6.9 MB/s | 283 kB 00:00 2020-11-16T21:27:53.133 INFO:teuthology.orchestra.run.smithi120.stdout:(28/36): python3-simplegeneric-0.8.1-17.el8.noa 921 kB/s | 19 kB 00:00 2020-11-16T21:27:53.151 INFO:teuthology.orchestra.run.smithi120.stdout:(29/36): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2020-11-16T21:27:53.153 INFO:teuthology.orchestra.run.smithi120.stdout:(30/36): python3-tempora-1.14.1-5.el8.noarch.rp 1.4 MB/s | 29 kB 00:00 2020-11-16T21:27:53.172 INFO:teuthology.orchestra.run.smithi120.stdout:(31/36): python3-trustme-0.6.0-4.el8.noarch.rpm 1.3 MB/s | 27 kB 00:00 2020-11-16T21:27:53.175 INFO:teuthology.orchestra.run.smithi120.stdout:(32/36): python3-waitress-1.2.1-2.el8.1.noarch. 11 MB/s | 241 kB 00:00 2020-11-16T21:27:53.193 INFO:teuthology.orchestra.run.smithi120.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 11 MB/s | 251 kB 00:00 2020-11-16T21:27:53.195 INFO:teuthology.orchestra.run.smithi120.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.1 MB/s | 85 kB 00:00 2020-11-16T21:27:53.213 INFO:teuthology.orchestra.run.smithi120.stdout:(35/36): python3-zc-lockfile-2.0-2.el8.noarch.r 1.1 MB/s | 23 kB 00:00 2020-11-16T21:27:53.356 INFO:teuthology.orchestra.run.smithi120.stdout:(36/36): ceph-osd-16.0.0-7219.g911cc357.el8.x86 16 MB/s | 18 MB 00:01 2020-11-16T21:27:53.360 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:27:53.361 INFO:teuthology.orchestra.run.smithi120.stdout:Total 21 MB/s | 31 MB 00:01 2020-11-16T21:27:53.373 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:27:53.415 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:27:53.415 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:27:53.755 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:27:53.755 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:27:54.947 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:27:55.158 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/36 2020-11-16T21:27:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/36 2020-11-16T21:27:55.574 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/36 2020-11-16T21:27:55.793 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/36 2020-11-16T21:27:55.942 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 5/36 2020-11-16T21:27:56.106 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/36 2020-11-16T21:27:56.262 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/36 2020-11-16T21:27:56.446 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/36 2020-11-16T21:27:56.582 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/36 2020-11-16T21:27:56.786 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/36 2020-11-16T21:27:56.978 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 11/36 2020-11-16T21:27:57.127 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 12/36 2020-11-16T21:27:57.292 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 13/36 2020-11-16T21:27:57.493 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-11-16T21:27:57.774 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-11-16T21:27:57.989 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-11-16T21:27:58.253 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-11-16T21:27:58.446 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-11-16T21:27:58.645 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-11-16T21:27:58.838 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-11-16T21:27:59.171 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-11-16T21:27:59.408 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-11-16T21:27:59.574 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-11-16T21:27:59.728 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-11-16T21:27:59.898 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-11-16T21:28:00.124 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-11-16T21:28:00.330 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-11-16T21:28:00.537 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-11-16T21:28:00.834 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-11-16T21:28:01.101 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-11-16T21:28:01.435 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-7219.g911cc357.el8. 31/36 2020-11-16T21:28:01.584 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:16.0.0-7219.g911cc357.el8.x86_64 32/36 2020-11-16T21:28:03.045 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:16.0.0-7219.g911cc357.el8.x86_64 32/36 2020-11-16T21:28:03.183 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:16.0.0-7219.g911cc357.el8.x86_64 33/36 2020-11-16T21:28:03.604 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:16.0.0-7219.g911cc357.el8.x86_64 33/36 2020-11-16T21:28:03.775 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:16.0.0-7219.g911cc357.el8.x86_64 34/36 2020-11-16T21:28:03.965 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:16.0.0-7219.g911cc357.el8.x86_64 34/36 2020-11-16T21:28:04.101 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:16.0.0-7219.g911cc357.el8.x86_64 35/36 2020-11-16T21:28:04.119 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:16.0.0-7219.g911cc357.el8.x86_64 35/36 2020-11-16T21:28:04.232 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:16.0.0-7219.g911cc357.el8.x86_64 36/36 2020-11-16T21:28:04.544 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-2:16.0.0-7219.g911cc357.el8.x86_64 36/36 2020-11-16T21:28:04.544 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:16.0.0-7219.g911cc357.el8.x86_64 1/36 2020-11-16T21:28:04.545 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:16.0.0-7219.g911cc357.el8.x86_64 2/36 2020-11-16T21:28:04.545 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:16.0.0-7219.g911cc357.el8.x86_64 3/36 2020-11-16T21:28:04.545 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:16.0.0-7219.g911cc357.el8.x86_64 4/36 2020-11-16T21:28:04.545 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:16.0.0-7219.g911cc357.el8.x86_64 5/36 2020-11-16T21:28:04.545 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-7219.g911cc357.el8. 6/36 2020-11-16T21:28:04.545 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-11-16T21:28:04.545 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-11-16T21:28:04.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-11-16T21:28:04.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-11-16T21:28:04.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-11-16T21:28:04.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-11-16T21:28:04.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-11-16T21:28:04.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-11-16T21:28:04.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-11-16T21:28:04.547 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-11-16T21:28:04.547 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-11-16T21:28:04.547 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/36 2020-11-16T21:28:04.547 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/36 2020-11-16T21:28:04.547 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 20/36 2020-11-16T21:28:04.547 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 21/36 2020-11-16T21:28:04.547 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 22/36 2020-11-16T21:28:04.548 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 23/36 2020-11-16T21:28:04.548 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 24/36 2020-11-16T21:28:04.548 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 25/36 2020-11-16T21:28:04.549 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 26/36 2020-11-16T21:28:04.549 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 27/36 2020-11-16T21:28:04.549 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-2.6-1.el8.noarch 28/36 2020-11-16T21:28:04.549 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 29/36 2020-11-16T21:28:04.549 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 30/36 2020-11-16T21:28:04.550 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 31/36 2020-11-16T21:28:04.550 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 32/36 2020-11-16T21:28:04.550 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 33/36 2020-11-16T21:28:04.550 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 34/36 2020-11-16T21:28:04.550 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 35/36 2020-11-16T21:28:05.199 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 36/36 2020-11-16T21:28:05.200 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:05.200 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:05.200 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:05.200 INFO:teuthology.orchestra.run.smithi120.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-11-16T21:28:05.200 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-11-16T21:28:05.200 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:16.0.0-7219.g911cc357.el8.noarch 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.0.6-13.el8.noarch 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-11-16T21:28:05.201 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2017.2-9.el8.noarch 2020-11-16T21:28:05.202 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-11-16T21:28:05.202 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-11-16T21:28:05.202 INFO:teuthology.orchestra.run.smithi120.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-11-16T21:28:05.202 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-11-16T21:28:05.202 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-11-16T21:28:05.202 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-6.2-6.el8.noarch 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-11-16T21:28:05.203 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-11-16T21:28:05.204 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-11-16T21:28:05.204 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-2.6-1.el8.noarch 2020-11-16T21:28:05.204 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-11-16T21:28:05.204 INFO:teuthology.orchestra.run.smithi120.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-11-16T21:28:05.204 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-11-16T21:28:05.204 INFO:teuthology.orchestra.run.smithi120.stdout: python3-trustme-0.6.0-4.el8.noarch 2020-11-16T21:28:05.204 INFO:teuthology.orchestra.run.smithi120.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-11-16T21:28:05.205 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-11-16T21:28:05.205 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-11-16T21:28:05.205 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-11-16T21:28:05.205 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:05.205 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:05.499 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base 2020-11-16T21:28:05.915 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:06.175 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-base-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:06.289 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:06.290 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:06.290 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:06.419 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install cephadm 2020-11-16T21:28:06.826 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:07.189 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:07.190 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:07.190 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2020-11-16T21:28:07.191 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:07.191 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:07.191 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm x86_64 2:16.0.0-7219.g911cc357.el8 ceph 57 k 2020-11-16T21:28:07.191 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:07.192 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:07.192 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:07.192 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-11-16T21:28:07.192 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:07.193 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 57 k 2020-11-16T21:28:07.193 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 218 k 2020-11-16T21:28:07.193 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:07.336 INFO:teuthology.orchestra.run.smithi120.stdout:cephadm-16.0.0-7219.g911cc357.el8.x86_64.rpm 394 kB/s | 57 kB 00:00 2020-11-16T21:28:07.336 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:07.337 INFO:teuthology.orchestra.run.smithi120.stdout:Total 391 kB/s | 57 kB 00:00 2020-11-16T21:28:07.338 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:07.345 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:07.346 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:07.347 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:07.348 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:07.391 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:09.203 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:09.336 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:09.486 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:09.747 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:09.747 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:09.747 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:09.747 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:09.747 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:09.748 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:09.995 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-immutable-object-cache 2020-11-16T21:28:10.419 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:10.790 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:10.790 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:10.791 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-11-16T21:28:10.791 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:10.791 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:10.791 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:16.0.0-7219.g911cc357.el8 ceph 201 k 2020-11-16T21:28:10.791 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:10.791 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:10.791 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:10.792 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-11-16T21:28:10.792 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:10.792 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 201 k 2020-11-16T21:28:10.792 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 584 k 2020-11-16T21:28:10.792 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:10.984 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-immutable-object-cache-16.0.0-7219.g911cc3 1.0 MB/s | 201 kB 00:00 2020-11-16T21:28:10.984 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:10.984 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.0 MB/s | 201 kB 00:00 2020-11-16T21:28:10.986 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:10.999 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:10.999 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:11.045 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:11.046 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:11.167 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:11.308 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-7219.g911cc357. 1/1 2020-11-16T21:28:11.625 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-7219.g911cc357. 1/1 2020-11-16T21:28:11.854 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-7219.g911cc357. 1/1 2020-11-16T21:28:11.854 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:11.855 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:11.855 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:11.855 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:11.855 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:12.060 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr 2020-11-16T21:28:12.477 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:12.732 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-mgr-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:12.844 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:12.845 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:12.845 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:12.966 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-dashboard 2020-11-16T21:28:13.376 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:13.764 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:13.765 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-11-16T21:28:13.765 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:13.765 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:16.0.0-7219.g911cc357.el8 ceph-noarch 3.2 M 2020-11-16T21:28:13.766 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:28:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:16.0.0-7219.g911cc357.el8 ceph-noarch 24 k 2020-11-16T21:28:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:16.0.0-7219.g911cc357.el8 ceph-noarch 8.8 k 2020-11-16T21:28:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2020-11-16T21:28:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2020-11-16T21:28:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2020-11-16T21:28:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:13.767 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:13.767 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:13.767 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2020-11-16T21:28:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:13.768 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 3.5 M 2020-11-16T21:28:13.768 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 14 M 2020-11-16T21:28:13.768 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:13.877 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): ceph-prometheus-alerts-16.0.0-7219.g911c 80 kB/s | 8.8 kB 00:00 2020-11-16T21:28:13.891 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): ceph-grafana-dashboards-16.0.0-7219.g911 194 kB/s | 24 kB 00:00 2020-11-16T21:28:13.908 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2020-11-16T21:28:13.941 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 676 kB/s | 34 kB 00:00 2020-11-16T21:28:14.024 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.7 MB/s | 196 kB 00:00 2020-11-16T21:28:14.089 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-mgr-dashboard-16.0.0-7219.g911cc357 9.9 MB/s | 3.2 MB 00:00 2020-11-16T21:28:14.090 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:14.091 INFO:teuthology.orchestra.run.smithi120.stdout:Total 11 MB/s | 3.5 MB 00:00 2020-11-16T21:28:14.094 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:14.103 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:14.104 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:14.256 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:14.257 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:14.520 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2020-11-16T21:28:14.917 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2020-11-16T21:28:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2020-11-16T21:28:15.247 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-7219.g911cc357.el8.n 4/6 2020-11-16T21:28:16.245 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-7219.g911cc357.el8. 5/6 2020-11-16T21:28:16.481 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-7219.g911cc357.el8.noarc 6/6 2020-11-16T21:28:16.686 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-7219.g911cc357.el8.noarc 6/6 2020-11-16T21:28:16.687 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-7219.g911cc357.el8. 1/6 2020-11-16T21:28:16.687 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-7219.g911cc357.el8.noarc 2/6 2020-11-16T21:28:16.687 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-7219.g911cc357.el8.n 3/6 2020-11-16T21:28:16.687 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2020-11-16T21:28:16.687 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2020-11-16T21:28:16.956 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2020-11-16T21:28:16.956 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:16.956 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:16.956 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:16.0.0-7219.g911cc357.el8.noarch 2020-11-16T21:28:16.956 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:16.0.0-7219.g911cc357.el8.noarch 2020-11-16T21:28:16.957 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:16.0.0-7219.g911cc357.el8.noarch 2020-11-16T21:28:16.957 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-11-16T21:28:16.957 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-11-16T21:28:16.957 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.4.1-12.el8.noarch 2020-11-16T21:28:16.957 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:16.957 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:17.085 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-diskprediction-local 2020-11-16T21:28:17.497 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:11 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:17.879 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:17.881 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:17.881 INFO:teuthology.orchestra.run.smithi120.stdout: Package ArchVersion Repository Size 2020-11-16T21:28:17.881 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:17.881 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:17.881 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch2:16.0.0-7219.g911cc357.el8 ceph-noarch 8.6 M 2020-11-16T21:28:17.881 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:28:17.882 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-11-16T21:28:17.882 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-11-16T21:28:17.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-11-16T21:28:17.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-11-16T21:28:17.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-11-16T21:28:17.882 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-11-16T21:28:17.882 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-11-16T21:28:17.883 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:17.883 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:17.883 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:17.883 INFO:teuthology.orchestra.run.smithi120.stdout:Install 8 Packages 2020-11-16T21:28:17.883 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:17.884 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 37 M 2020-11-16T21:28:17.884 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 205 M 2020-11-16T21:28:17.884 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:17.998 INFO:teuthology.orchestra.run.smithi120.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 37 MB/s | 4.3 MB 00:00 2020-11-16T21:28:18.017 INFO:teuthology.orchestra.run.smithi120.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2020-11-16T21:28:18.024 INFO:teuthology.orchestra.run.smithi120.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 30 MB/s | 225 kB 00:00 2020-11-16T21:28:18.076 INFO:teuthology.orchestra.run.smithi120.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 48 MB/s | 3.7 MB 00:00 2020-11-16T21:28:18.091 INFO:teuthology.orchestra.run.smithi120.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 43 MB/s | 638 kB 00:00 2020-11-16T21:28:18.097 INFO:teuthology.orchestra.run.smithi120.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 26 MB/s | 169 kB 00:00 2020-11-16T21:28:18.309 INFO:teuthology.orchestra.run.smithi120.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 51 MB/s | 15 MB 00:00 2020-11-16T21:28:18.415 INFO:teuthology.orchestra.run.smithi120.stdout:(8/8): ceph-mgr-diskprediction-local-16.0.0-721 16 MB/s | 8.6 MB 00:00 2020-11-16T21:28:18.417 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:18.418 INFO:teuthology.orchestra.run.smithi120.stdout:Total 68 MB/s | 37 MB 00:00 2020-11-16T21:28:18.428 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:18.446 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:18.446 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:18.748 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:18.749 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:19.286 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:20.881 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-11-16T21:28:21.281 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-11-16T21:28:21.497 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-11-16T21:28:22.041 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-11-16T21:28:22.206 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-11-16T21:28:22.749 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-11-16T21:28:22.939 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-11-16T21:28:23.625 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-11-16T21:28:23.926 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-11-16T21:28:24.130 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-11-16T21:28:26.074 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-11-16T21:28:27.338 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-11-16T21:28:27.539 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-7219.g911cc35 8/8 2020-11-16T21:28:27.783 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-7219.g911cc35 8/8 2020-11-16T21:28:27.784 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-7219.g911cc35 1/8 2020-11-16T21:28:27.784 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-11-16T21:28:27.784 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-11-16T21:28:27.784 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-11-16T21:28:27.784 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-11-16T21:28:27.784 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-11-16T21:28:27.785 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-11-16T21:28:28.054 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-11-16T21:28:28.054 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:28.055 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:28.055 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:16.0.0-7219.g911cc357.el8.noarch 2020-11-16T21:28:28.055 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.3-2.el8.x86_64 2020-11-16T21:28:28.055 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-11-16T21:28:28.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-11-16T21:28:28.056 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-11-16T21:28:28.056 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-11-16T21:28:28.056 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-11-16T21:28:28.056 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-11-16T21:28:28.056 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:28.057 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:28.179 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-rook 2020-11-16T21:28:28.590 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:22 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:28.963 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:28.965 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:28.965 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-11-16T21:28:28.965 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:28.965 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:28.965 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:16.0.0-7219.g911cc357.el8 2020-11-16T21:28:28.966 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-noarch 27 k 2020-11-16T21:28:28.966 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:28:28.966 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-11-16T21:28:28.966 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-11-16T21:28:28.966 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-11-16T21:28:28.966 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-11-16T21:28:28.966 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-11-16T21:28:28.967 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-11-16T21:28:28.967 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-11-16T21:28:28.967 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-11-16T21:28:28.967 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-11-16T21:28:28.967 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 11.0.0-2.el8 epel 1.2 M 2020-11-16T21:28:28.967 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 3.4.2-15.el8 epel 73 k 2020-11-16T21:28:28.967 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-11-16T21:28:28.968 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:28.968 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:28.968 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:28.968 INFO:teuthology.orchestra.run.smithi120.stdout:Install 13 Packages 2020-11-16T21:28:28.968 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:28.969 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.9 M 2020-11-16T21:28:28.969 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 24 M 2020-11-16T21:28:28.969 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:29.002 INFO:teuthology.orchestra.run.smithi120.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 786 kB/s | 27 kB 00:00 2020-11-16T21:28:29.005 INFO:teuthology.orchestra.run.smithi120.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 47 MB/s | 126 kB 00:00 2020-11-16T21:28:29.006 INFO:teuthology.orchestra.run.smithi120.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 529 kB/s | 20 kB 00:00 2020-11-16T21:28:29.007 INFO:teuthology.orchestra.run.smithi120.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 42 MB/s | 110 kB 00:00 2020-11-16T21:28:29.009 INFO:teuthology.orchestra.run.smithi120.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 44 MB/s | 155 kB 00:00 2020-11-16T21:28:29.010 INFO:teuthology.orchestra.run.smithi120.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 18 MB/s | 43 kB 00:00 2020-11-16T21:28:29.069 INFO:teuthology.orchestra.run.smithi120.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 278 kB/s | 16 kB 00:00 2020-11-16T21:28:29.069 INFO:teuthology.orchestra.run.smithi120.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 555 kB/s | 33 kB 00:00 2020-11-16T21:28:29.090 INFO:teuthology.orchestra.run.smithi120.stdout:(9/13): ceph-mgr-rook-16.0.0-7219.g911cc357.el8 222 kB/s | 27 kB 00:00 2020-11-16T21:28:29.108 INFO:teuthology.orchestra.run.smithi120.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 2.0 MB/s | 81 kB 00:00 2020-11-16T21:28:29.128 INFO:teuthology.orchestra.run.smithi120.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.0 MB/s | 61 kB 00:00 2020-11-16T21:28:29.155 INFO:teuthology.orchestra.run.smithi120.stdout:(12/13): python3-rsa-3.4.2-15.el8.noarch.rpm 1.1 MB/s | 73 kB 00:00 2020-11-16T21:28:29.206 INFO:teuthology.orchestra.run.smithi120.stdout:(13/13): python3-kubernetes-11.0.0-2.el8.noarch 8.5 MB/s | 1.2 MB 00:00 2020-11-16T21:28:29.207 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:29.208 INFO:teuthology.orchestra.run.smithi120.stdout:Total 8.0 MB/s | 1.9 MB 00:00 2020-11-16T21:28:29.214 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:29.228 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:29.228 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:29.316 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:29.317 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:29.767 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:30.050 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-11-16T21:28:30.268 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-11-16T21:28:30.465 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-3.4.2-15.el8.noarch 3/13 2020-11-16T21:28:30.635 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-11-16T21:28:30.793 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-11-16T21:28:31.016 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-11-16T21:28:31.298 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-11-16T21:28:31.510 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-11-16T21:28:33.069 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-11-16T21:28:33.287 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 10/13 2020-11-16T21:28:33.453 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-11-16T21:28:33.645 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-11-16T21:28:33.788 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:16.0.0-7219.g911cc357.el8.noarch 13/13 2020-11-16T21:28:34.007 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-7219.g911cc357.el8.noarch 13/13 2020-11-16T21:28:34.008 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:16.0.0-7219.g911cc357.el8.noarch 1/13 2020-11-16T21:28:34.008 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-11-16T21:28:34.008 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-11-16T21:28:34.008 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-11-16T21:28:34.008 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-11-16T21:28:34.009 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-11-16T21:28:34.009 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-11-16T21:28:34.009 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-11-16T21:28:34.009 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-11-16T21:28:34.009 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-11-16T21:28:34.009 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 11/13 2020-11-16T21:28:34.009 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-3.4.2-15.el8.noarch 12/13 2020-11-16T21:28:34.318 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-11-16T21:28:34.318 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:34.319 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:34.319 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:16.0.0-7219.g911cc357.el8.noarch 2020-11-16T21:28:34.319 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-11-16T21:28:34.319 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-11-16T21:28:34.319 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-11-16T21:28:34.319 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-11-16T21:28:34.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-11-16T21:28:34.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-11-16T21:28:34.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-11-16T21:28:34.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-11-16T21:28:34.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-11-16T21:28:34.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-11-16T21:28:34.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-3.4.2-15.el8.noarch 2020-11-16T21:28:34.321 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-11-16T21:28:34.321 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:34.321 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:34.464 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-cephadm 2020-11-16T21:28:34.875 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:35.250 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:35.251 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:35.251 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-11-16T21:28:35.251 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:35.251 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:35.252 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:16.0.0-7219.g911cc357.el8 ceph-noarch 66 k 2020-11-16T21:28:35.252 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:28:35.252 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2020-11-16T21:28:35.252 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2020-11-16T21:28:35.252 INFO:teuthology.orchestra.run.smithi120.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2020-11-16T21:28:35.252 INFO:teuthology.orchestra.run.smithi120.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2020-11-16T21:28:35.253 INFO:teuthology.orchestra.run.smithi120.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2020-11-16T21:28:35.253 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:35.253 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:35.253 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:35.253 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2020-11-16T21:28:35.253 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:35.254 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 5.8 M 2020-11-16T21:28:35.254 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 24 M 2020-11-16T21:28:35.254 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:35.296 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 12 MB/s | 538 kB 00:00 2020-11-16T21:28:35.344 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 53 MB/s | 4.8 MB 00:00 2020-11-16T21:28:35.355 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 360 kB/s | 21 kB 00:00 2020-11-16T21:28:35.394 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.2 MB/s | 48 kB 00:00 2020-11-16T21:28:35.400 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): ceph-mgr-cephadm-16.0.0-7219.g911cc357.e 451 kB/s | 66 kB 00:00 2020-11-16T21:28:35.442 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.5 MB/s | 355 kB 00:00 2020-11-16T21:28:35.444 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:35.445 INFO:teuthology.orchestra.run.smithi120.stdout:Total 30 MB/s | 5.8 MB 00:00 2020-11-16T21:28:35.449 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:35.457 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:35.458 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:35.537 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:35.538 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:35.780 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:36.070 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2020-11-16T21:28:36.289 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2020-11-16T21:28:37.198 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2020-11-16T21:28:37.831 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2020-11-16T21:28:38.055 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2020-11-16T21:28:38.195 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-7219.g911cc357.el8.noarch 6/6 2020-11-16T21:28:38.405 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-7219.g911cc357.el8.noarch 6/6 2020-11-16T21:28:38.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-7219.g911cc357.el8.noarch 1/6 2020-11-16T21:28:38.406 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2020-11-16T21:28:38.406 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2020-11-16T21:28:38.406 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2020-11-16T21:28:38.406 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2020-11-16T21:28:38.660 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2020-11-16T21:28:38.661 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:38.661 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:38.661 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:16.0.0-7219.g911cc357.el8.noarch 2020-11-16T21:28:38.661 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.5.1-5.el8.noarch 2020-11-16T21:28:38.661 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2020-11-16T21:28:38.662 INFO:teuthology.orchestra.run.smithi120.stdout: python3-apipkg-1.5-6.el8.noarch 2020-11-16T21:28:38.662 INFO:teuthology.orchestra.run.smithi120.stdout: python3-execnet-1.7.1-1.el8.noarch 2020-11-16T21:28:38.662 INFO:teuthology.orchestra.run.smithi120.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-11-16T21:28:38.662 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:38.662 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:38.798 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-fuse 2020-11-16T21:28:39.212 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:39.599 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:39.600 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:39.600 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-11-16T21:28:39.600 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:39.600 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:39.600 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:16.0.0-7219.g911cc357.el8 ceph 624 k 2020-11-16T21:28:39.601 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-11-16T21:28:39.601 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-11-16T21:28:39.601 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-11-16T21:28:39.601 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:39.601 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:39.601 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:39.601 INFO:teuthology.orchestra.run.smithi120.stdout:Install 3 Packages 2020-11-16T21:28:39.602 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:39.602 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 728 k 2020-11-16T21:28:39.602 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.9 M 2020-11-16T21:28:39.602 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:39.635 INFO:teuthology.orchestra.run.smithi120.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 82 kB 00:00 2020-11-16T21:28:39.635 INFO:teuthology.orchestra.run.smithi120.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 632 kB/s | 21 kB 00:00 2020-11-16T21:28:39.819 INFO:teuthology.orchestra.run.smithi120.stdout:(3/3): ceph-fuse-16.0.0-7219.g911cc357.el8.x86_ 2.8 MB/s | 624 kB 00:00 2020-11-16T21:28:39.820 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:39.820 INFO:teuthology.orchestra.run.smithi120.stdout:Total 3.2 MB/s | 728 kB 00:00 2020-11-16T21:28:39.824 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:39.845 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:39.845 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:39.909 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:39.910 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:40.074 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:40.074 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-11-16T21:28:40.074 INFO:teuthology.orchestra.run.smithi120.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-11-16T21:28:40.075 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:40.437 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-11-16T21:28:40.577 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 3/3 2020-11-16T21:28:40.862 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 3/3 2020-11-16T21:28:40.862 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 1/3 2020-11-16T21:28:40.862 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-11-16T21:28:41.135 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-11-16T21:28:41.136 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:41.136 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:41.136 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-11-16T21:28:41.136 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-11-16T21:28:41.136 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:41.136 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:41.430 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install librados-devel 2020-11-16T21:28:41.850 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:42.270 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:42.271 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:42.271 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-11-16T21:28:42.271 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:16.0.0-7219.g911cc357.el8 ceph 131 k 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:42.272 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 131 k 2020-11-16T21:28:42.273 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 430 k 2020-11-16T21:28:42.273 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:42.441 INFO:teuthology.orchestra.run.smithi120.stdout:librados-devel-16.0.0-7219.g911cc357.el8.x86_64 774 kB/s | 131 kB 00:00 2020-11-16T21:28:42.442 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:42.442 INFO:teuthology.orchestra.run.smithi120.stdout:Total 769 kB/s | 131 kB 00:00 2020-11-16T21:28:42.443 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:42.450 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:42.450 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:42.498 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:42.499 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:42.633 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:43.184 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:43.399 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados-devel-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:43.641 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:43.642 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:43.642 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:43.642 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:43.642 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:43.642 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:43.947 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs2 2020-11-16T21:28:44.375 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:44.635 INFO:teuthology.orchestra.run.smithi120.stdout:Package libcephfs2-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:44.750 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:44.751 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:44.752 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:44.876 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs-devel 2020-11-16T21:28:45.289 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:45.681 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:45.682 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:45.683 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-11-16T21:28:45.683 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:45.683 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:45.683 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:16.0.0-7219.g911cc357.el8 ceph 24 k 2020-11-16T21:28:45.683 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:45.683 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:45.684 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:45.684 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-11-16T21:28:45.684 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:45.684 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 24 k 2020-11-16T21:28:45.684 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 87 k 2020-11-16T21:28:45.684 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:45.807 INFO:teuthology.orchestra.run.smithi120.stdout:libcephfs-devel-16.0.0-7219.g911cc357.el8.x86_6 191 kB/s | 24 kB 00:00 2020-11-16T21:28:45.807 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:45.808 INFO:teuthology.orchestra.run.smithi120.stdout:Total 190 kB/s | 24 kB 00:00 2020-11-16T21:28:45.809 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:45.813 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:45.813 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:45.816 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:45.816 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:45.868 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:46.000 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:46.174 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs-devel-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:46.382 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:46.383 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:46.383 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:46.383 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:46.383 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:46.383 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:46.513 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install librados2 2020-11-16T21:28:46.927 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:47.185 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:47.299 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:47.300 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:47.300 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:47.425 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install librbd1 2020-11-16T21:28:47.842 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:48.101 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:48.215 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:48.216 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:48.216 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:48.345 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rados 2020-11-16T21:28:48.760 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:49.018 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rados-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:49.132 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:49.133 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:49.133 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:49.258 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rgw 2020-11-16T21:28:49.676 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:49.943 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rgw-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:50.057 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:50.058 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:50.058 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:50.194 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-cephfs 2020-11-16T21:28:50.609 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:50.884 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-cephfs-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:50.997 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:50.998 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:50.998 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:51.141 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rbd 2020-11-16T21:28:51.555 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:51.825 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rbd-2:16.0.0-7219.g911cc357.el8.x86_64 is already installed. 2020-11-16T21:28:51.940 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:51.941 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-11-16T21:28:51.941 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:52.078 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-fuse 2020-11-16T21:28:52.492 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:52.857 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:52.858 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:52.858 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2020-11-16T21:28:52.858 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:52.858 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:52.858 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:16.0.0-7219.g911cc357.el8 ceph 87 k 2020-11-16T21:28:52.858 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:52.859 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:52.859 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:52.859 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-11-16T21:28:52.859 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:52.859 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 87 k 2020-11-16T21:28:52.859 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 236 k 2020-11-16T21:28:52.859 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:53.027 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-fuse-16.0.0-7219.g911cc357.el8.x86_64.rpm 515 kB/s | 87 kB 00:00 2020-11-16T21:28:53.028 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:53.028 INFO:teuthology.orchestra.run.smithi120.stdout:Total 512 kB/s | 87 kB 00:00 2020-11-16T21:28:53.029 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:53.040 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:53.041 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:53.089 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:53.089 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:53.201 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:53.341 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:53.529 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:54.424 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:54.425 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:55.261 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-mirror 2020-11-16T21:28:55.688 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:56.090 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:56.091 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:56.091 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-11-16T21:28:56.092 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:56.092 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:56.092 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:16.0.0-7219.g911cc357.el8 ceph 3.1 M 2020-11-16T21:28:56.092 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:56.092 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:56.092 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:56.092 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-11-16T21:28:56.093 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:56.093 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 3.1 M 2020-11-16T21:28:56.093 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 11 M 2020-11-16T21:28:56.093 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-mirror-16.0.0-7219.g911cc357.el8.x86_64.rpm 10 MB/s | 3.1 MB 00:00 2020-11-16T21:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout:Total 10 MB/s | 3.1 MB 00:00 2020-11-16T21:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:56.481 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:56.481 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:56.862 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:57.034 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:57.351 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:57.583 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:57.583 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:57.583 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:57.583 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:57.583 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:57.584 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:57.870 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-nbd 2020-11-16T21:28:58.290 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Nov 2020 09:27:06 PM UTC. 2020-11-16T21:28:58.655 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-11-16T21:28:58.656 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:58.656 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2020-11-16T21:28:58.656 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:58.656 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-11-16T21:28:58.657 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:16.0.0-7219.g911cc357.el8 ceph 168 k 2020-11-16T21:28:58.657 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:58.657 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-11-16T21:28:58.657 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-11-16T21:28:58.657 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-11-16T21:28:58.657 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:58.657 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 168 k 2020-11-16T21:28:58.658 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 503 k 2020-11-16T21:28:58.658 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-11-16T21:28:58.827 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-nbd-16.0.0-7219.g911cc357.el8.x86_64.rpm 992 kB/s | 168 kB 00:00 2020-11-16T21:28:58.827 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-11-16T21:28:58.828 INFO:teuthology.orchestra.run.smithi120.stdout:Total 986 kB/s | 168 kB 00:00 2020-11-16T21:28:58.829 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-11-16T21:28:58.841 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-11-16T21:28:58.841 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-11-16T21:28:58.890 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-11-16T21:28:58.890 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-11-16T21:28:59.019 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-11-16T21:28:59.189 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:59.401 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-nbd-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:59.657 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:16.0.0-7219.g911cc357.el8.x86_64 1/1 2020-11-16T21:28:59.657 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:59.657 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-11-16T21:28:59.658 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:16.0.0-7219.g911cc357.el8.x86_64 2020-11-16T21:28:59.658 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:28:59.658 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-11-16T21:28:59.946 DEBUG:teuthology.parallel:result is None 2020-11-16T21:28:59.947 INFO:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2020-11-16T21:28:59.984 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS 2020-11-16T21:28:59.985 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-16T21:28:59.986 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T21:29:00.150 INFO:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-11-16T21:29:00.169 INFO:teuthology.orchestra.run.smithi120.stdout:16.0.0-7219.g911cc357.el8 2020-11-16T21:29:00.170 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7219.g911cc357.el8 2020-11-16T21:29:00.171 INFO:teuthology.task.install:The correct ceph version 16.0.0-7219.g911cc357 is installed. 2020-11-16T21:29:00.171 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-16T21:29:00.171 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:00.172 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-16T21:29:00.222 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-16T21:29:00.235 INFO:teuthology.orchestra.run.smithi120.stderr:24+1 records in 2020-11-16T21:29:00.235 INFO:teuthology.orchestra.run.smithi120.stderr:24+1 records out 2020-11-16T21:29:00.236 INFO:teuthology.orchestra.run.smithi120.stderr:12667 bytes (13 kB, 12 KiB) copied, 7.2237e-05 s, 175 MB/s 2020-11-16T21:29:00.236 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-16T21:29:00.237 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:00.237 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2020-11-16T21:29:00.289 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-16T21:29:00.302 INFO:teuthology.orchestra.run.smithi120.stderr:5+1 records in 2020-11-16T21:29:00.302 INFO:teuthology.orchestra.run.smithi120.stderr:5+1 records out 2020-11-16T21:29:00.302 INFO:teuthology.orchestra.run.smithi120.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 3.419e-05 s, 89.4 MB/s 2020-11-16T21:29:00.303 INFO:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-16T21:29:00.370 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-16T21:29:00.370 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:00.371 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-16T21:29:00.422 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-16T21:29:00.435 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:00.436 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:00.436 INFO:teuthology.orchestra.run.smithi120.stderr:299 bytes copied, 2.7059e-05 s, 11.0 MB/s 2020-11-16T21:29:00.437 INFO:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-16T21:29:00.503 INFO:teuthology.run_tasks:Running task ceph... 2020-11-16T21:29:00.578 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-16T21:29:00.580 INFO:teuthology.orchestra.run.smithi120:> sudo chmod 777 /var/log/ceph 2020-11-16T21:29:00.607 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-16T21:29:00.608 INFO:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-16T21:29:00.674 INFO:tasks.ceph:Creating extra log directories... 2020-11-16T21:29:00.675 INFO:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-16T21:29:00.776 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-11-16T21:29:00.776 INFO:tasks.ceph:config {'conf': {'osd': {'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'Health detail', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2020-11-16T21:29:00.777 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154', 'branch': 'wip-kefu-testing-2020-11-16-1845', 'description': 'rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}', 'email': 'tchaikov@gmail.com', 'first_in_suite': False, 'job_id': '5631154', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.1', 'overrides': {'admin_socket': {'branch': 'wip-kefu-testing-2020-11-16-1845'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'Health detail', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'Health detail'], 'mon_bind_msgr2': False, 'sha1': '911cc35794d871071a8bebfb0919482263b1d18b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '911cc35794d871071a8bebfb0919482263b1d18b'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-kefu-testing-2020-11-16-1845', 'sha1': '911cc35794d871071a8bebfb0919482263b1d18b'}}, 'owner': 'scheduled_kchai@teuthology', 'priority': 90, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2']], 'sha1': '911cc35794d871071a8bebfb0919482263b1d18b', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-kefu-testing-2020-11-16-1845', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-16-1845/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '911cc35794d871071a8bebfb0919482263b1d18b', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (1, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'Health detail', '\\(OSD_SLOW_PING_TIME'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'Health detail'], 'mon_bind_msgr2': False, 'sha1': '911cc35794d871071a8bebfb0919482263b1d18b', 'cluster': 'ceph'}}, {'osd_backfill': None}], 'teuthology_branch': 'master', 'timestamp': '2020-11-16_12:11:03', 'tube': 'smithi', 'user': 'kchai', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.11580', 'targets': {'smithi120.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgazuGy1FDOixf3HJOVsflIrbMzrTtxieWw4kKWyrMzsKquLnoPrQYrZBPCNWnRVBhde1lsPzVjsUZosfJkvpsV2FX9JZSP03t2LX05FEoXUoIev8npix8/GyOrHNszuT9k+4xPcOsgDbtuvUZ1LMExV783IKeXxYx2+w3YUkehGLxKPzDyN+Cf2DOYkTL7JPnUQASEOmZdOql5sF14IQcOhYD1CEPxeN6OfI6ndrjs6bmCcLveFYsfRJluA3FPMVa3s/ogL6+dFSFdA7vGHUpDjwCLv1P5obSJNfOiDXbpgaDyjs9juKXgAg2tlTHemn2glyCIyRFmrbvViuza9QiRZzOirDbIIAhyhGzC49g9ePxF+MrRRkJ58hy8WUuVOKBn9DLew2D3Y6yXnhfwHriNMz+XcSBV32xQ11V/aBVcQj+aZGaaKmkvAzBjrWiolpTw+knOvuP3y42vZYz4pfwcMjmAZHLmMvuNoIzE04dj/VwKIxJAvsCfq/HiGUWLCU='}} 2020-11-16T21:29:00.777 INFO:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-11-16T21:29:00.798 INFO:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/run/ceph 2020-11-16T21:29:00.865 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:00.866 INFO:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2020-11-16T21:29:00.918 INFO:teuthology.orchestra.run.smithi120.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-16T21:29:00.919 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_1 2020-11-16T21:29:00.919 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_2 2020-11-16T21:29:00.920 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_3 2020-11-16T21:29:00.920 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_4 2020-11-16T21:29:00.920 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:00.920 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:00.920 INFO:teuthology.orchestra.run.smithi120.stderr:72 bytes copied, 2.9303e-05 s, 2.5 MB/s 2020-11-16T21:29:00.920 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-16T21:29:00.921 INFO:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-16T21:29:00.973 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-16T21:29:00.973 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-16T21:29:00.974 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 62828 Links: 1 2020-11-16T21:29:00.974 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-16T21:29:00.974 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2020-11-16T21:29:00.974 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2020-11-16 21:28:57.196815358 +0000 2020-11-16T21:29:00.974 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2020-11-16 21:26:27.004949181 +0000 2020-11-16T21:29:00.975 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2020-11-16 21:26:27.004949181 +0000 2020-11-16T21:29:00.975 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2020-11-16T21:29:00.987 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2020-11-16T21:29:00.987 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2020-11-16T21:29:00.987 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000221487 s, 2.3 MB/s 2020-11-16T21:29:00.989 INFO:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-16T21:29:01.043 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-16T21:29:01.044 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-16T21:29:01.044 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 67645 Links: 1 2020-11-16T21:29:01.044 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-16T21:29:01.044 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2020-11-16T21:29:01.044 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2020-11-16 21:28:57.196815358 +0000 2020-11-16T21:29:01.044 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2020-11-16 21:26:31.413848245 +0000 2020-11-16T21:29:01.044 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2020-11-16 21:26:31.413848245 +0000 2020-11-16T21:29:01.045 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2020-11-16T21:29:01.056 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2020-11-16T21:29:01.056 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2020-11-16T21:29:01.056 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000206636 s, 2.5 MB/s 2020-11-16T21:29:01.058 INFO:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-16T21:29:01.112 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-16T21:29:01.113 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-16T21:29:01.113 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 64979 Links: 1 2020-11-16T21:29:01.113 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-16T21:29:01.113 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2020-11-16T21:29:01.113 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2020-11-16 21:28:57.196815358 +0000 2020-11-16T21:29:01.113 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2020-11-16 21:26:39.907653783 +0000 2020-11-16T21:29:01.114 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2020-11-16 21:26:39.907653783 +0000 2020-11-16T21:29:01.114 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2020-11-16T21:29:01.125 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2020-11-16T21:29:01.126 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2020-11-16T21:29:01.126 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000224354 s, 2.3 MB/s 2020-11-16T21:29:01.128 INFO:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-16T21:29:01.181 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-16T21:29:01.182 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-16T21:29:01.182 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 67711 Links: 1 2020-11-16T21:29:01.182 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-16T21:29:01.182 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2020-11-16T21:29:01.182 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2020-11-16 21:28:57.196815358 +0000 2020-11-16T21:29:01.182 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2020-11-16 21:26:40.217646684 +0000 2020-11-16T21:29:01.183 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2020-11-16 21:26:40.217646684 +0000 2020-11-16T21:29:01.183 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2020-11-16T21:29:01.194 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2020-11-16T21:29:01.195 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2020-11-16T21:29:01.195 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000257512 s, 2.0 MB/s 2020-11-16T21:29:01.197 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2020-11-16T21:29:01.197 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-16T21:29:01.197 INFO:tasks.ceph:Generating config... 2020-11-16T21:29:01.199 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2020-11-16T21:29:01.199 INFO:tasks.ceph:[osd] bdev async discard = True 2020-11-16T21:29:01.199 INFO:tasks.ceph:[osd] bdev enable discard = True 2020-11-16T21:29:01.200 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2020-11-16T21:29:01.200 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2020-11-16T21:29:01.200 INFO:tasks.ceph:[osd] debug bluefs = 20 2020-11-16T21:29:01.200 INFO:tasks.ceph:[osd] debug bluestore = 20 2020-11-16T21:29:01.200 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-16T21:29:01.201 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-16T21:29:01.201 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-16T21:29:01.201 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-16T21:29:01.201 INFO:tasks.ceph:[osd] debug rocksdb = 10 2020-11-16T21:29:01.201 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2020-11-16T21:29:01.202 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2020-11-16T21:29:01.202 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2020-11-16T21:29:01.202 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2020-11-16T21:29:01.202 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2020-11-16T21:29:01.202 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2020-11-16T21:29:01.203 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2020-11-16T21:29:01.203 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2020-11-16T21:29:01.203 INFO:tasks.ceph:[osd] osd op queue = debug_random 2020-11-16T21:29:01.203 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2020-11-16T21:29:01.203 INFO:tasks.ceph:[global] mon client directed command retry = 5 2020-11-16T21:29:01.204 INFO:tasks.ceph:[global] mon election default strategy = 1 2020-11-16T21:29:01.204 INFO:tasks.ceph:[global] ms bind msgr2 = False 2020-11-16T21:29:01.204 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2020-11-16T21:29:01.204 INFO:tasks.ceph:[global] ms type = async 2020-11-16T21:29:01.204 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-16T21:29:01.205 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-16T21:29:01.205 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-16T21:29:01.205 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-16T21:29:01.205 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-16T21:29:01.206 INFO:tasks.ceph:Setting up mon.a... 2020-11-16T21:29:01.206 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-11-16T21:29:01.353 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.keyring 2020-11-16T21:29:01.355 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-11-16T21:29:01.395 INFO:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-16T21:29:01.462 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.120:6789]'), ('mon.b', '[v1:172.21.15.120:6790]'), ('mon.c', '[v1:172.21.15.120:6791]')] 2020-11-16T21:29:01.463 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.120:6789],[v1:172.21.15.120:6790],[v1:172.21.15.120:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2020-11-16T21:29:01.464 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:01.464 INFO:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-16T21:29:01.515 INFO:teuthology.orchestra.run.smithi120.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-16T21:29:01.516 INFO:teuthology.orchestra.run.smithi120.stderr:6+1 records in 2020-11-16T21:29:01.516 INFO:teuthology.orchestra.run.smithi120.stderr:6+1 records out 2020-11-16T21:29:01.516 INFO:teuthology.orchestra.run.smithi120.stderr:3274 bytes (3.3 kB, 3.2 KiB) copied, 3.6618e-05 s, 89.4 MB/s 2020-11-16T21:29:01.516 INFO:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.120:6789]' --addv b '[v1:172.21.15.120:6790]' --addv c '[v1:172.21.15.120:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:01.595 INFO:teuthology.orchestra.run.smithi120.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:01.595 INFO:teuthology.orchestra.run.smithi120.stdout:monmaptool: generated fsid 29829f84-fc87-4495-9b6d-a546578bfa29 2020-11-16T21:29:01.595 INFO:teuthology.orchestra.run.smithi120.stdout:epoch 0 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:fsid 29829f84-fc87-4495-9b6d-a546578bfa29 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:last_changed 2020-11-16T21:29:01.604364+0000 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:created 2020-11-16T21:29:01.604364+0000 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:min_mon_release 0 (unknown) 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:election_strategy: 1 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:0: v1:172.21.15.120:6789/0 mon.a 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:1: v1:172.21.15.120:6790/0 mon.b 2020-11-16T21:29:01.596 INFO:teuthology.orchestra.run.smithi120.stdout:2: v1:172.21.15.120:6791/0 mon.c 2020-11-16T21:29:01.597 INFO:teuthology.orchestra.run.smithi120.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-11-16T21:29:01.598 INFO:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-16T21:29:01.650 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 29829f84-fc87-4495-9b6d-a546578bfa29... 2020-11-16T21:29:01.651 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-16T21:29:01.742 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2020-11-16T21:29:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2020-11-16T21:29:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-16T21:29:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2020-11-16T21:29:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2020-11-16T21:29:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-16T21:29:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2020-11-16T21:29:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2020-11-16T21:29:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-16T21:29:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2020-11-16T21:29:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2020-11-16T21:29:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2020-11-16T21:29:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2020-11-16T21:29:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2020-11-16T21:29:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2020-11-16T21:29:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2020-11-16T21:29:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2020-11-16T21:29:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2020-11-16T21:29:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2020-11-16T21:29:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2020-11-16T21:29:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2020-11-16T21:29:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "[v1:172.21.15.120:6789],[v1:172.21.15.120:6790],[v1:172.21.15.120:6791]" 2020-11-16T21:29:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: mon client directed command retry = 5 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: mon election default strategy = 1 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: ms bind msgr2 = False 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: ms inject socket failures = 5000 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: ms type = async 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = 29829f84-fc87-4495-9b6d-a546578bfa29 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2020-11-16T21:29:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2020-11-16T21:29:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2020-11-16T21:29:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2020-11-16T21:29:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2020-11-16T21:29:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2020-11-16T21:29:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2020-11-16T21:29:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2020-11-16T21:29:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2020-11-16T21:29:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2020-11-16T21:29:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: osd min pg log entries = 5 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2020-11-16T21:29:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2020-11-16T21:29:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 20 2020-11-16T21:29:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 20 2020-11-16T21:29:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: debug filestore = 20 2020-11-16T21:29:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: debug journal = 20 2020-11-16T21:29:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 20 2020-11-16T21:29:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 25 2020-11-16T21:29:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 10 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify cached snaps = True 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify missing on start = True 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: osd memory target = 1610612736 2020-11-16T21:29:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: osd op queue = debug_random 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: osd op queue cut off = debug_random 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2020-11-16T21:29:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2020-11-16T21:29:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2020-11-16T21:29:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2020-11-16T21:29:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2020-11-16T21:29:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.757 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2020-11-16T21:29:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2020-11-16T21:29:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2020-11-16T21:29:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2020-11-16T21:29:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2020-11-16T21:29:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2020-11-16T21:29:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2020-11-16T21:29:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2020-11-16T21:29:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2020-11-16T21:29:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-16T21:29:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-16T21:29:01.760 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2020-11-16T21:29:01.760 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2020-11-16T21:29:01.760 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2020-11-16T21:29:01.761 INFO:tasks.ceph:Creating admin key on mon.a... 2020-11-16T21:29:01.761 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-11-16T21:29:01.812 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-16T21:29:01.813 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:01.813 INFO:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-16T21:29:01.825 INFO:teuthology.orchestra.run.smithi120.stderr:+ dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-16T21:29:01.826 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.] 2020-11-16T21:29:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAd77JfLeb8FxAAfT/UNEJnERy1pEXbA05Ajw== 2020-11-16T21:29:01.827 INFO:teuthology.orchestra.run.smithi120.stdout:[client.admin] 2020-11-16T21:29:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAd77JfNMnXMBAAViiDHYegzV7A8nY/n0W7lA== 2020-11-16T21:29:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: caps mds = "allow *" 2020-11-16T21:29:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: caps mgr = "allow *" 2020-11-16T21:29:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: caps mon = "allow *" 2020-11-16T21:29:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: caps osd = "allow *" 2020-11-16T21:29:01.827 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:01.828 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:01.828 INFO:teuthology.orchestra.run.smithi120.stderr:206 bytes copied, 3.6482e-05 s, 5.6 MB/s 2020-11-16T21:29:01.828 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:01.828 INFO:teuthology.orchestra.run.smithi120:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-16T21:29:01.878 INFO:teuthology.orchestra.run.smithi120.stderr:+ dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-16T21:29:01.879 INFO:teuthology.orchestra.run.smithi120.stdout: *)�����D��m�FW��)�_��$�_��$�a5a��xb5b��xc5c��xabc 2020-11-16T21:29:01.880 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:01.880 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:01.880 INFO:teuthology.orchestra.run.smithi120.stderr:304 bytes copied, 3.078e-05 s, 9.9 MB/s 2020-11-16T21:29:01.880 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:29:01.880 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:01.881 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.keyring 2020-11-16T21:29:01.881 INFO:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-16T21:29:01.931 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/etc/ceph/ceph.keyring 2020-11-16T21:29:01.944 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:01.944 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:01.945 INFO:teuthology.orchestra.run.smithi120.stderr:206 bytes copied, 5.6126e-05 s, 3.7 MB/s 2020-11-16T21:29:01.945 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-16T21:29:01.958 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:01.959 INFO:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:02.011 INFO:teuthology.orchestra.run.smithi120.stderr:+ dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:02.012 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:02.012 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:02.012 INFO:teuthology.orchestra.run.smithi120.stderr:304 bytes copied, 4.1909e-05 s, 7.3 MB/s 2020-11-16T21:29:02.012 INFO:tasks.ceph:Setting up mon nodes... 2020-11-16T21:29:02.013 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-16T21:29:02.013 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-11-16T21:29:02.105 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-11-16T21:29:02.107 INFO:tasks.ceph:Setting up mds nodes... 2020-11-16T21:29:02.107 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-16T21:29:02.107 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-16T21:29:02.108 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-16T21:29:02.108 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-11-16T21:29:02.135 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2020-11-16T21:29:02.136 INFO:tasks.ceph:role: osd.0 2020-11-16T21:29:02.136 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:29:02.136 INFO:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-16T21:29:02.666 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-16T21:29:02.667 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-16T21:29:02.667 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-16T21:29:02.667 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 2020-11-16T21:29:02.667 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-16T21:29:02.667 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2020-11-16T21:29:02.667 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-16T21:29:02.668 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-16T21:29:02.668 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-16T21:29:02.668 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-16T21:29:02.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2020-11-16T21:29:02.669 INFO:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2020-11-16T21:29:02.777 INFO:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-11-16T21:29:02.808 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-11-16T21:29:02.875 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2020-11-16T21:29:02.875 INFO:tasks.ceph:role: osd.1 2020-11-16T21:29:02.875 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:29:02.875 INFO:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-16T21:29:03.393 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-16T21:29:03.393 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-16T21:29:03.393 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-16T21:29:03.393 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 2020-11-16T21:29:03.394 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-16T21:29:03.394 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2020-11-16T21:29:03.394 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-16T21:29:03.394 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-16T21:29:03.394 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-16T21:29:03.394 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-16T21:29:03.395 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2020-11-16T21:29:03.396 INFO:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2020-11-16T21:29:03.445 INFO:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-11-16T21:29:03.476 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-11-16T21:29:03.541 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2020-11-16T21:29:03.542 INFO:tasks.ceph:role: osd.2 2020-11-16T21:29:03.542 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:29:03.542 INFO:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-16T21:29:04.060 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-16T21:29:04.061 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-16T21:29:04.061 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-16T21:29:04.061 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 2020-11-16T21:29:04.061 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-16T21:29:04.061 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2020-11-16T21:29:04.062 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-16T21:29:04.062 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-16T21:29:04.062 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-16T21:29:04.062 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-16T21:29:04.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2020-11-16T21:29:04.063 INFO:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2020-11-16T21:29:04.136 INFO:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-11-16T21:29:04.167 INFO:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:04.252 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:04.259+0000 7f60c012eec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-11-16T21:29:04.252 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:04.259+0000 7f60c012eec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-11-16T21:29:04.252 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:04.259+0000 7f60c012eec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-16T21:29:04.253 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:04.259+0000 7f60c012eec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-16T21:29:04.253 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:04.259+0000 7f60c012eec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-16T21:29:04.253 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:04.260+0000 7f60c012eec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2020-11-16T21:29:07.103 INFO:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-11-16T21:29:07.134 INFO:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:07.223 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:07.230+0000 7f672945dec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-11-16T21:29:07.223 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:07.230+0000 7f672945dec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-11-16T21:29:07.223 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:07.230+0000 7f672945dec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-16T21:29:07.224 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:07.230+0000 7f672945dec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-16T21:29:07.224 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:07.230+0000 7f672945dec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-16T21:29:07.224 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:07.231+0000 7f672945dec0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2020-11-16T21:29:10.073 INFO:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-11-16T21:29:10.105 INFO:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:10.193 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:10.201+0000 7fad38e0eec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-11-16T21:29:10.193 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:10.201+0000 7fad38e0eec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-11-16T21:29:10.193 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:10.201+0000 7fad38e0eec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-16T21:29:10.194 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:10.201+0000 7fad38e0eec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-16T21:29:10.194 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:10.201+0000 7fad38e0eec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-16T21:29:10.194 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:10.202+0000 7fad38e0eec0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2020-11-16T21:29:13.037 INFO:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-11-16T21:29:13.069 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-16T21:29:13.069 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:13.070 INFO:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-16T21:29:13.124 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-16T21:29:13.137 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2020-11-16T21:29:13.138 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAe77Jff0bQBhAAdCqxpumPGoD78a86nYhriw== 2020-11-16T21:29:13.138 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:13.138 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:13.138 INFO:teuthology.orchestra.run.smithi120.stderr:56 bytes copied, 3.6601e-05 s, 1.5 MB/s 2020-11-16T21:29:13.139 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:13.139 INFO:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-16T21:29:13.193 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-16T21:29:13.207 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2020-11-16T21:29:13.207 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAg77Jf5RGQDxAAZMky6SAGgLEDJPI3gujEKQ== 2020-11-16T21:29:13.208 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:13.208 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:13.208 INFO:teuthology.orchestra.run.smithi120.stderr:56 bytes copied, 3.4265e-05 s, 1.6 MB/s 2020-11-16T21:29:13.209 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:13.210 INFO:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-16T21:29:13.265 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-16T21:29:13.278 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2020-11-16T21:29:13.278 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAj77JfNOzUDRAAQwBagdSalCwiEK4qVOxRDA== 2020-11-16T21:29:13.279 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:13.279 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:13.279 INFO:teuthology.orchestra.run.smithi120.stderr:56 bytes copied, 3.7093e-05 s, 1.5 MB/s 2020-11-16T21:29:13.280 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:13.280 INFO:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-16T21:29:13.333 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-16T21:29:13.346 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2020-11-16T21:29:13.346 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAm77JfEyIJDBAAOp4V+0l/I1z+VaAPl7A6cA== 2020-11-16T21:29:13.346 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:13.346 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:13.347 INFO:teuthology.orchestra.run.smithi120.stderr:56 bytes copied, 4.8298e-05 s, 1.2 MB/s 2020-11-16T21:29:13.347 INFO:tasks.ceph:Adding keys to all mons... 2020-11-16T21:29:13.348 INFO:teuthology.orchestra.run.smithi120:> sudo tee -a /etc/ceph/ceph.keyring 2020-11-16T21:29:13.413 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2020-11-16T21:29:13.414 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAe77Jff0bQBhAAdCqxpumPGoD78a86nYhriw== 2020-11-16T21:29:13.414 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2020-11-16T21:29:13.414 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAg77Jf5RGQDxAAZMky6SAGgLEDJPI3gujEKQ== 2020-11-16T21:29:13.414 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2020-11-16T21:29:13.414 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAj77JfNOzUDRAAQwBagdSalCwiEK4qVOxRDA== 2020-11-16T21:29:13.414 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2020-11-16T21:29:13.415 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAm77JfEyIJDBAAOp4V+0l/I1z+VaAPl7A6cA== 2020-11-16T21:29:13.415 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-16T21:29:13.504 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-16T21:29:13.552 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-16T21:29:13.600 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-16T21:29:13.648 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-16T21:29:13.648 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-11-16T21:29:13.674 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-16T21:29:13.767 INFO:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-11-16T21:29:13.793 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-11-16T21:29:13.860 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-16T21:29:13.954 INFO:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-11-16T21:29:13.981 INFO:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-11-16T21:29:14.047 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-16T21:29:14.141 INFO:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-11-16T21:29:14.168 INFO:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-11-16T21:29:14.221 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-11-16T21:29:14.222 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-16T21:29:14.222 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-16T21:29:14.263 INFO:tasks.ceph.mon.a:Started 2020-11-16T21:29:14.264 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-16T21:29:14.265 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-16T21:29:14.266 INFO:tasks.ceph.mon.b:Started 2020-11-16T21:29:14.267 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-16T21:29:14.267 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-16T21:29:14.269 INFO:tasks.ceph.mon.c:Started 2020-11-16T21:29:14.269 INFO:teuthology.orchestra.run.smithi120:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2020-11-16T21:29:24.501 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-11-16T21:29:24.502 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-11-16T21:29:24.503 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-11-16T21:29:24.505 INFO:tasks.ceph.mgr.x:Started 2020-11-16T21:29:24.505 INFO:tasks.ceph:Setting crush tunables to default 2020-11-16T21:29:24.506 INFO:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd crush tunables default 2020-11-16T21:29:24.721 INFO:teuthology.orchestra.run.smithi120.stderr:adjusted tunables profile to default 2020-11-16T21:29:24.731 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-11-16T21:29:24.732 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:24.732 INFO:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-16T21:29:24.744 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-16T21:29:24.758 INFO:teuthology.orchestra.run.smithi120.stdout:27509b0a-c110-47b7-8822-cf9defe592ac 2020-11-16T21:29:24.758 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:24.759 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:24.759 INFO:teuthology.orchestra.run.smithi120.stderr:37 bytes copied, 3.9691e-05 s, 932 kB/s 2020-11-16T21:29:24.759 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:24.760 INFO:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-16T21:29:24.812 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-16T21:29:24.825 INFO:teuthology.orchestra.run.smithi120.stdout:77627eca-b580-425e-b593-30753d001ffc 2020-11-16T21:29:24.825 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:24.825 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:24.825 INFO:teuthology.orchestra.run.smithi120.stderr:37 bytes copied, 3.0085e-05 s, 1.2 MB/s 2020-11-16T21:29:24.826 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-11-16T21:29:24.826 INFO:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-16T21:29:24.879 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-16T21:29:24.893 INFO:teuthology.orchestra.run.smithi120.stdout:05547d39-7932-479e-9303-fc0320537140 2020-11-16T21:29:24.893 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-11-16T21:29:24.893 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-11-16T21:29:24.893 INFO:teuthology.orchestra.run.smithi120.stderr:37 bytes copied, 3.5817e-05 s, 1.0 MB/s 2020-11-16T21:29:24.894 INFO:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 27509b0a-c110-47b7-8822-cf9defe592ac 0 2020-11-16T21:29:25.157 INFO:teuthology.orchestra.run.smithi120.stdout:0 2020-11-16T21:29:25.169 INFO:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 77627eca-b580-425e-b593-30753d001ffc 1 2020-11-16T21:29:25.392 INFO:teuthology.orchestra.run.smithi120.stdout:1 2020-11-16T21:29:25.410 INFO:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 05547d39-7932-479e-9303-fc0320537140 2 2020-11-16T21:29:25.628 INFO:teuthology.orchestra.run.smithi120.stdout:2 2020-11-16T21:29:25.640 INFO:tasks.ceph.osd.0:Restarting daemon 2020-11-16T21:29:25.640 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-11-16T21:29:25.642 INFO:tasks.ceph.osd.0:Started 2020-11-16T21:29:25.642 INFO:tasks.ceph.osd.1:Restarting daemon 2020-11-16T21:29:25.643 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-11-16T21:29:25.644 INFO:tasks.ceph.osd.1:Started 2020-11-16T21:29:25.645 INFO:tasks.ceph.osd.2:Restarting daemon 2020-11-16T21:29:25.645 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-11-16T21:29:25.647 INFO:tasks.ceph.osd.2:Started 2020-11-16T21:29:25.647 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-16T21:29:25.949 INFO:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-16T21:29:26.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:29:26.024+0000 7f54cb38aec0 -1 Falling back to public interface 2020-11-16T21:29:26.024 INFO:tasks.ceph.osd.2.smithi120.stderr:2020-11-16T21:29:26.032+0000 7f363351fec0 -1 Falling back to public interface 2020-11-16T21:29:26.026 INFO:tasks.ceph.osd.0.smithi120.stderr:2020-11-16T21:29:26.034+0000 7f83b6b07ec0 -1 Falling back to public interface 2020-11-16T21:29:26.137 INFO:teuthology.misc.health.smithi120.stdout: 2020-11-16T21:29:26.138 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":5,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:29:25.635252+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:29:26.148 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-16T21:29:27.338 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:29:27.347+0000 7f54cb38aec0 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:29:27.360 INFO:tasks.ceph.osd.0.smithi120.stderr:2020-11-16T21:29:27.368+0000 7f83b6b07ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:29:27.360 INFO:tasks.ceph.osd.2.smithi120.stderr:2020-11-16T21:29:27.368+0000 7f363351fec0 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:29:28.407 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:29:28.415+0000 7f54c1b9c700 -1 osd.1 0 waiting for initial osdmap 2020-11-16T21:29:28.410 INFO:tasks.ceph.osd.0.smithi120.stderr:2020-11-16T21:29:28.418+0000 7f83ad319700 -1 osd.0 0 waiting for initial osdmap 2020-11-16T21:29:28.412 INFO:tasks.ceph.osd.2.smithi120.stderr:2020-11-16T21:29:28.420+0000 7f362b534700 -1 osd.2 0 waiting for initial osdmap 2020-11-16T21:29:32.452 INFO:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-16T21:29:32.740 INFO:teuthology.misc.health.smithi120.stdout: 2020-11-16T21:29:32.741 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":9,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:29:31.370198+0000","last_up_change":"2020-11-16T21:29:30.365578+0000","last_in_change":"2020-11-16T21:29:30.365578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":21984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":21984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":21984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":21984}]},"public_addr":"172.21.15.120:6808/21984","cluster_addr":"172.21.15.120:6809/21984","heartbeat_back_addr":"172.21.15.120:6811/21984","heartbeat_front_addr":"172.21.15.120:6810/21984","state":["exists","up"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":21985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":21985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":21985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":21985}]},"public_addr":"172.21.15.120:6800/21985","cluster_addr":"172.21.15.120:6801/21985","heartbeat_back_addr":"172.21.15.120:6803/21985","heartbeat_front_addr":"172.21.15.120:6802/21985","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:29:32.752 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-16T21:29:32.752 INFO:tasks.ceph:Creating RBD pool 2020-11-16T21:29:32.753 INFO:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd pool create rbd 8 2020-11-16T21:29:33.581 INFO:teuthology.orchestra.run.smithi120.stderr:pool 'rbd' created 2020-11-16T21:29:33.593 INFO:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-16T21:29:34.592 INFO:teuthology.orchestra.run.smithi120.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-16T21:29:34.606 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-11-16T21:29:34.607 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:29:34.607 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-16T21:29:34.897 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:34.898 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":11,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:29:34.583888+0000","last_up_change":"2020-11-16T21:29:30.365578+0000","last_in_change":"2020-11-16T21:29:30.365578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":21984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":21984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":21984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":21984}]},"public_addr":"172.21.15.120:6808/21984","cluster_addr":"172.21.15.120:6809/21984","heartbeat_back_addr":"172.21.15.120:6811/21984","heartbeat_front_addr":"172.21.15.120:6810/21984","state":["exists","up"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":21985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":21985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":21985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":21985}]},"public_addr":"172.21.15.120:6800/21985","cluster_addr":"172.21.15.120:6801/21985","heartbeat_back_addr":"172.21.15.120:6803/21985","heartbeat_front_addr":"172.21.15.120:6802/21985","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:29:34.909 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2020-11-16T21:29:33.080349+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-16T21:29:34.910 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-16T21:29:35.198 INFO:teuthology.orchestra.run.smithi120.stdout:pg_num: 8 2020-11-16T21:29:35.210 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-11-16T21:29:35.211 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-11-16T21:29:35.211 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2020-11-16T21:29:35.544 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:35.555 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":4,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6812","nonce":21697}]},"active_addr":"172.21.15.120:6812/21697","active_change":"2020-11-16T21:29:28.038818+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":3036539820}]},{"addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":374987847}]}]}} 2020-11-16T21:29:35.558 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-11-16T21:29:35.558 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-11-16T21:29:35.558 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:29:35.853 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:35.853 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":12,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:29:35.587828+0000","last_up_change":"2020-11-16T21:29:30.365578+0000","last_in_change":"2020-11-16T21:29:30.365578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":21984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":21984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":21984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":21984}]},"public_addr":"172.21.15.120:6808/21984","cluster_addr":"172.21.15.120:6809/21984","heartbeat_back_addr":"172.21.15.120:6811/21984","heartbeat_front_addr":"172.21.15.120:6810/21984","state":["exists","up"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":21985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":21985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":21985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":21985}]},"public_addr":"172.21.15.120:6800/21985","cluster_addr":"172.21.15.120:6801/21985","heartbeat_back_addr":"172.21.15.120:6803/21985","heartbeat_front_addr":"172.21.15.120:6802/21985","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:29:35.865 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-11-16T21:29:35.866 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:29:36.153 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:36.153 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":12,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:29:35.587828+0000","last_up_change":"2020-11-16T21:29:30.365578+0000","last_in_change":"2020-11-16T21:29:30.365578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":21984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":21984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":21984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":21984}]},"public_addr":"172.21.15.120:6808/21984","cluster_addr":"172.21.15.120:6809/21984","heartbeat_back_addr":"172.21.15.120:6811/21984","heartbeat_front_addr":"172.21.15.120:6810/21984","state":["exists","up"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":21985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":21985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":21985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":21985}]},"public_addr":"172.21.15.120:6800/21985","cluster_addr":"172.21.15.120:6801/21985","heartbeat_back_addr":"172.21.15.120:6803/21985","heartbeat_front_addr":"172.21.15.120:6802/21985","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:29:36.165 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2020-11-16T21:29:36.315 INFO:teuthology.orchestra.run.smithi120.stdout:34359738372 2020-11-16T21:29:36.315 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-16T21:29:36.465 INFO:teuthology.orchestra.run.smithi120.stdout:34359738371 2020-11-16T21:29:36.466 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-16T21:29:36.617 INFO:teuthology.orchestra.run.smithi120.stdout:34359738372 2020-11-16T21:29:36.617 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-16T21:29:36.906 INFO:teuthology.orchestra.run.smithi120.stdout:34359738370 2020-11-16T21:29:36.918 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738370 for osd.0 2020-11-16T21:29:37.920 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-16T21:29:38.214 INFO:teuthology.orchestra.run.smithi120.stdout:34359738370 2020-11-16T21:29:38.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738370 for osd.0 2020-11-16T21:29:39.227 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-16T21:29:39.519 INFO:teuthology.orchestra.run.smithi120.stdout:34359738372 2020-11-16T21:29:39.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2020-11-16T21:29:39.531 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:29:39.819 INFO:teuthology.orchestra.run.smithi120.stdout:34359738371 2020-11-16T21:29:39.831 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2020-11-16T21:29:39.831 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-16T21:29:40.118 INFO:teuthology.orchestra.run.smithi120.stdout:34359738372 2020-11-16T21:29:40.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2020-11-16T21:29:40.131 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-11-16T21:29:40.131 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:29:40.416 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:29:40.428 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-11-16T21:29:40.043864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14728,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100792,"statfs":{"total":289910292480,"available":289895211008,"internally_reserved":0,"allocated":311296,"data_stored":70147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607622+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:29:34.607622+0000","last_peered":"2020-11-16T21:29:34.607622+0000","last_clean":"2020-11-16T21:29:34.607622+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:29:34.607622+0000","last_undegraded":"2020-11-16T21:29:34.607622+0000","last_fullsized":"2020-11-16T21:29:34.607622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.608057+0000","last_change":"2020-11-16T21:29:34.606262+0000","last_active":"2020-11-16T21:29:34.608057+0000","last_peered":"2020-11-16T21:29:34.608057+0000","last_clean":"2020-11-16T21:29:34.608057+0000","last_became_active":"2020-11-16T21:29:34.605864+0000","last_became_peered":"2020-11-16T21:29:34.605864+0000","last_unstale":"2020-11-16T21:29:34.608057+0000","last_undegraded":"2020-11-16T21:29:34.608057+0000","last_fullsized":"2020-11-16T21:29:34.608057+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.610836+0000","last_change":"2020-11-16T21:29:34.609971+0000","last_active":"2020-11-16T21:29:34.610836+0000","last_peered":"2020-11-16T21:29:34.610836+0000","last_clean":"2020-11-16T21:29:34.610836+0000","last_became_active":"2020-11-16T21:29:34.609635+0000","last_became_peered":"2020-11-16T21:29:34.609635+0000","last_unstale":"2020-11-16T21:29:34.610836+0000","last_undegraded":"2020-11-16T21:29:34.610836+0000","last_fullsized":"2020-11-16T21:29:34.610836+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607721+0000","last_change":"2020-11-16T21:29:34.606142+0000","last_active":"2020-11-16T21:29:34.607721+0000","last_peered":"2020-11-16T21:29:34.607721+0000","last_clean":"2020-11-16T21:29:34.607721+0000","last_became_active":"2020-11-16T21:29:34.604646+0000","last_became_peered":"2020-11-16T21:29:34.604646+0000","last_unstale":"2020-11-16T21:29:34.607721+0000","last_undegraded":"2020-11-16T21:29:34.607721+0000","last_fullsized":"2020-11-16T21:29:34.607721+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"8","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.608554+0000","last_change":"2020-11-16T21:29:34.608554+0000","last_active":"2020-11-16T21:29:34.608554+0000","last_peered":"2020-11-16T21:29:34.608554+0000","last_clean":"2020-11-16T21:29:34.608554+0000","last_became_active":"2020-11-16T21:29:34.608237+0000","last_became_peered":"2020-11-16T21:29:34.608237+0000","last_unstale":"2020-11-16T21:29:34.608554+0000","last_undegraded":"2020-11-16T21:29:34.608554+0000","last_fullsized":"2020-11-16T21:29:34.608554+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.602975+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:29:34.602975+0000","last_peered":"2020-11-16T21:29:34.602975+0000","last_clean":"2020-11-16T21:29:34.602975+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:29:34.602975+0000","last_undegraded":"2020-11-16T21:29:34.602975+0000","last_fullsized":"2020-11-16T21:29:34.602975+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607679+0000","last_change":"2020-11-16T21:29:34.604960+0000","last_active":"2020-11-16T21:29:34.607679+0000","last_peered":"2020-11-16T21:29:34.607679+0000","last_clean":"2020-11-16T21:29:34.607679+0000","last_became_active":"2020-11-16T21:29:34.604467+0000","last_became_peered":"2020-11-16T21:29:34.604467+0000","last_unstale":"2020-11-16T21:29:34.607679+0000","last_undegraded":"2020-11-16T21:29:34.607679+0000","last_fullsized":"2020-11-16T21:29:34.607679+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.606900+0000","last_change":"2020-11-16T21:29:34.606067+0000","last_active":"2020-11-16T21:29:34.606900+0000","last_peered":"2020-11-16T21:29:34.606900+0000","last_clean":"2020-11-16T21:29:34.606900+0000","last_became_active":"2020-11-16T21:29:34.605672+0000","last_became_peered":"2020-11-16T21:29:34.605672+0000","last_unstale":"2020-11-16T21:29:34.606900+0000","last_undegraded":"2020-11-16T21:29:34.606900+0000","last_fullsized":"2020-11-16T21:29:34.606900+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":21587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]}]}} 2020-11-16T21:29:40.429 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:29:40.709 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:40.711 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:29:40.721 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-11-16T21:29:40.043864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14728,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100792,"statfs":{"total":289910292480,"available":289895211008,"internally_reserved":0,"allocated":311296,"data_stored":70147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607622+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:29:34.607622+0000","last_peered":"2020-11-16T21:29:34.607622+0000","last_clean":"2020-11-16T21:29:34.607622+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:29:34.607622+0000","last_undegraded":"2020-11-16T21:29:34.607622+0000","last_fullsized":"2020-11-16T21:29:34.607622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.608057+0000","last_change":"2020-11-16T21:29:34.606262+0000","last_active":"2020-11-16T21:29:34.608057+0000","last_peered":"2020-11-16T21:29:34.608057+0000","last_clean":"2020-11-16T21:29:34.608057+0000","last_became_active":"2020-11-16T21:29:34.605864+0000","last_became_peered":"2020-11-16T21:29:34.605864+0000","last_unstale":"2020-11-16T21:29:34.608057+0000","last_undegraded":"2020-11-16T21:29:34.608057+0000","last_fullsized":"2020-11-16T21:29:34.608057+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.610836+0000","last_change":"2020-11-16T21:29:34.609971+0000","last_active":"2020-11-16T21:29:34.610836+0000","last_peered":"2020-11-16T21:29:34.610836+0000","last_clean":"2020-11-16T21:29:34.610836+0000","last_became_active":"2020-11-16T21:29:34.609635+0000","last_became_peered":"2020-11-16T21:29:34.609635+0000","last_unstale":"2020-11-16T21:29:34.610836+0000","last_undegraded":"2020-11-16T21:29:34.610836+0000","last_fullsized":"2020-11-16T21:29:34.610836+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607721+0000","last_change":"2020-11-16T21:29:34.606142+0000","last_active":"2020-11-16T21:29:34.607721+0000","last_peered":"2020-11-16T21:29:34.607721+0000","last_clean":"2020-11-16T21:29:34.607721+0000","last_became_active":"2020-11-16T21:29:34.604646+0000","last_became_peered":"2020-11-16T21:29:34.604646+0000","last_unstale":"2020-11-16T21:29:34.607721+0000","last_undegraded":"2020-11-16T21:29:34.607721+0000","last_fullsized":"2020-11-16T21:29:34.607721+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"8","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.608554+0000","last_change":"2020-11-16T21:29:34.608554+0000","last_active":"2020-11-16T21:29:34.608554+0000","last_peered":"2020-11-16T21:29:34.608554+0000","last_clean":"2020-11-16T21:29:34.608554+0000","last_became_active":"2020-11-16T21:29:34.608237+0000","last_became_peered":"2020-11-16T21:29:34.608237+0000","last_unstale":"2020-11-16T21:29:34.608554+0000","last_undegraded":"2020-11-16T21:29:34.608554+0000","last_fullsized":"2020-11-16T21:29:34.608554+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.602975+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:29:34.602975+0000","last_peered":"2020-11-16T21:29:34.602975+0000","last_clean":"2020-11-16T21:29:34.602975+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:29:34.602975+0000","last_undegraded":"2020-11-16T21:29:34.602975+0000","last_fullsized":"2020-11-16T21:29:34.602975+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607679+0000","last_change":"2020-11-16T21:29:34.604960+0000","last_active":"2020-11-16T21:29:34.607679+0000","last_peered":"2020-11-16T21:29:34.607679+0000","last_clean":"2020-11-16T21:29:34.607679+0000","last_became_active":"2020-11-16T21:29:34.604467+0000","last_became_peered":"2020-11-16T21:29:34.604467+0000","last_unstale":"2020-11-16T21:29:34.607679+0000","last_undegraded":"2020-11-16T21:29:34.607679+0000","last_fullsized":"2020-11-16T21:29:34.607679+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.606900+0000","last_change":"2020-11-16T21:29:34.606067+0000","last_active":"2020-11-16T21:29:34.606900+0000","last_peered":"2020-11-16T21:29:34.606900+0000","last_clean":"2020-11-16T21:29:34.606900+0000","last_became_active":"2020-11-16T21:29:34.605672+0000","last_became_peered":"2020-11-16T21:29:34.605672+0000","last_unstale":"2020-11-16T21:29:34.606900+0000","last_undegraded":"2020-11-16T21:29:34.606900+0000","last_fullsized":"2020-11-16T21:29:34.606900+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":21587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]}]}} 2020-11-16T21:29:40.722 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-11-16T21:29:40.723 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-11-16T21:29:40.723 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-11-16T21:29:40.723 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2020-11-16T21:29:41.073 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:41.085 INFO:teuthology.orchestra.run.smithi120.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-16T21:29:41.086 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-11-16T21:29:41.086 INFO:teuthology.run_tasks:Running task osd_backfill... 2020-11-16T21:29:41.099 INFO:tasks.osd_backfill:num_osds is 3 2020-11-16T21:29:41.100 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:29:41.390 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:41.390 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":12,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:29:35.587828+0000","last_up_change":"2020-11-16T21:29:30.365578+0000","last_in_change":"2020-11-16T21:29:30.365578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":21984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":21984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":21984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":21984}]},"public_addr":"172.21.15.120:6808/21984","cluster_addr":"172.21.15.120:6809/21984","heartbeat_back_addr":"172.21.15.120:6811/21984","heartbeat_front_addr":"172.21.15.120:6810/21984","state":["exists","up"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":21985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":21985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":21985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":21985}]},"public_addr":"172.21.15.120:6800/21985","cluster_addr":"172.21.15.120:6801/21985","heartbeat_back_addr":"172.21.15.120:6803/21985","heartbeat_front_addr":"172.21.15.120:6802/21985","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:29:41.403 INFO:tasks.osd_backfill.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2020-11-16T21:29:33.080349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-16T21:29:41.403 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-16T21:29:41.693 INFO:teuthology.orchestra.run.smithi120.stdout:pg_num: 8 2020-11-16T21:29:41.705 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2020-11-16T21:29:41.992 INFO:teuthology.orchestra.run.smithi120.stdout:epoch 12 2020-11-16T21:29:41.992 INFO:teuthology.orchestra.run.smithi120.stdout:fsid 29829f84-fc87-4495-9b6d-a546578bfa29 2020-11-16T21:29:41.993 INFO:teuthology.orchestra.run.smithi120.stdout:created 2020-11-16T21:29:24.399586+0000 2020-11-16T21:29:41.993 INFO:teuthology.orchestra.run.smithi120.stdout:modified 2020-11-16T21:29:35.587828+0000 2020-11-16T21:29:41.993 INFO:teuthology.orchestra.run.smithi120.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2020-11-16T21:29:41.993 INFO:teuthology.orchestra.run.smithi120.stdout:crush_version 4 2020-11-16T21:29:41.993 INFO:teuthology.orchestra.run.smithi120.stdout:full_ratio 0.95 2020-11-16T21:29:41.993 INFO:teuthology.orchestra.run.smithi120.stdout:backfillfull_ratio 0.9 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:nearfull_ratio 0.85 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:require_min_compat_client luminous 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:min_compat_client jewel 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:require_osd_release pacific 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:stretch_mode_enabled false 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 12 flags hashpspool stripe_width 0 application rbd 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:max_osd 3 2020-11-16T21:29:41.994 INFO:teuthology.orchestra.run.smithi120.stdout:osd.0 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v1:172.21.15.120:6808/21984 v1:172.21.15.120:6809/21984 exists,up 27509b0a-c110-47b7-8822-cf9defe592ac 2020-11-16T21:29:41.995 INFO:teuthology.orchestra.run.smithi120.stdout:osd.1 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v1:172.21.15.120:6800/21985 v1:172.21.15.120:6801/21985 exists,up 77627eca-b580-425e-b593-30753d001ffc 2020-11-16T21:29:41.995 INFO:teuthology.orchestra.run.smithi120.stdout:osd.2 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v1:172.21.15.120:6804/21986 v1:172.21.15.120:6805/21986 exists,up 05547d39-7932-479e-9303-fc0320537140 2020-11-16T21:29:42.004 INFO:tasks.osd_backfill.ceph_manager:['osd.0 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v1:172.21.15.120:6808/21984 v1:172.21.15.120:6809/21984 exists,up 27509b0a-c110-47b7-8822-cf9defe592ac', 'osd.1 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v1:172.21.15.120:6800/21985 v1:172.21.15.120:6801/21985 exists,up 77627eca-b580-425e-b593-30753d001ffc', 'osd.2 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v1:172.21.15.120:6804/21986 v1:172.21.15.120:6805/21986 exists,up 05547d39-7932-479e-9303-fc0320537140'] 2020-11-16T21:29:42.005 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2020-11-16T21:29:42.155 INFO:teuthology.orchestra.run.smithi120.stdout:34359738374 2020-11-16T21:29:42.155 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-16T21:29:42.305 INFO:teuthology.orchestra.run.smithi120.stdout:34359738373 2020-11-16T21:29:42.305 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-16T21:29:42.459 INFO:teuthology.orchestra.run.smithi120.stdout:34359738374 2020-11-16T21:29:42.459 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-16T21:29:42.747 INFO:teuthology.orchestra.run.smithi120.stdout:34359738373 2020-11-16T21:29:42.759 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738374 got 34359738373 for osd.0 2020-11-16T21:29:43.761 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-16T21:29:44.057 INFO:teuthology.orchestra.run.smithi120.stdout:34359738373 2020-11-16T21:29:44.069 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738374 got 34359738373 for osd.0 2020-11-16T21:29:45.070 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-16T21:29:45.365 INFO:teuthology.orchestra.run.smithi120.stdout:34359738374 2020-11-16T21:29:45.378 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738374 got 34359738374 for osd.0 2020-11-16T21:29:45.378 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:29:45.666 INFO:teuthology.orchestra.run.smithi120.stdout:34359738373 2020-11-16T21:29:45.678 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738373 got 34359738373 for osd.1 2020-11-16T21:29:45.679 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-16T21:29:45.966 INFO:teuthology.orchestra.run.smithi120.stdout:34359738374 2020-11-16T21:29:45.978 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738374 got 34359738374 for osd.2 2020-11-16T21:29:45.978 INFO:tasks.osd_backfill.ceph_manager:waiting for clean 2020-11-16T21:29:45.978 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:29:46.262 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:46.264 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:29:46.274 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2020-11-16T21:29:46.044805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14736,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100784,"statfs":{"total":289910292480,"available":289895202816,"internally_reserved":0,"allocated":319488,"data_stored":72840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607622+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:29:34.607622+0000","last_peered":"2020-11-16T21:29:34.607622+0000","last_clean":"2020-11-16T21:29:34.607622+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:29:34.607622+0000","last_undegraded":"2020-11-16T21:29:34.607622+0000","last_fullsized":"2020-11-16T21:29:34.607622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.608057+0000","last_change":"2020-11-16T21:29:34.606262+0000","last_active":"2020-11-16T21:29:34.608057+0000","last_peered":"2020-11-16T21:29:34.608057+0000","last_clean":"2020-11-16T21:29:34.608057+0000","last_became_active":"2020-11-16T21:29:34.605864+0000","last_became_peered":"2020-11-16T21:29:34.605864+0000","last_unstale":"2020-11-16T21:29:34.608057+0000","last_undegraded":"2020-11-16T21:29:34.608057+0000","last_fullsized":"2020-11-16T21:29:34.608057+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.610836+0000","last_change":"2020-11-16T21:29:34.609971+0000","last_active":"2020-11-16T21:29:34.610836+0000","last_peered":"2020-11-16T21:29:34.610836+0000","last_clean":"2020-11-16T21:29:34.610836+0000","last_became_active":"2020-11-16T21:29:34.609635+0000","last_became_peered":"2020-11-16T21:29:34.609635+0000","last_unstale":"2020-11-16T21:29:34.610836+0000","last_undegraded":"2020-11-16T21:29:34.610836+0000","last_fullsized":"2020-11-16T21:29:34.610836+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607721+0000","last_change":"2020-11-16T21:29:34.606142+0000","last_active":"2020-11-16T21:29:34.607721+0000","last_peered":"2020-11-16T21:29:34.607721+0000","last_clean":"2020-11-16T21:29:34.607721+0000","last_became_active":"2020-11-16T21:29:34.604646+0000","last_became_peered":"2020-11-16T21:29:34.604646+0000","last_unstale":"2020-11-16T21:29:34.607721+0000","last_undegraded":"2020-11-16T21:29:34.607721+0000","last_fullsized":"2020-11-16T21:29:34.607721+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.610083+0000","last_change":"2020-11-16T21:29:34.608554+0000","last_active":"2020-11-16T21:29:34.610083+0000","last_peered":"2020-11-16T21:29:34.610083+0000","last_clean":"2020-11-16T21:29:34.610083+0000","last_became_active":"2020-11-16T21:29:34.608237+0000","last_became_peered":"2020-11-16T21:29:34.608237+0000","last_unstale":"2020-11-16T21:29:34.610083+0000","last_undegraded":"2020-11-16T21:29:34.610083+0000","last_fullsized":"2020-11-16T21:29:34.610083+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.602975+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:29:34.602975+0000","last_peered":"2020-11-16T21:29:34.602975+0000","last_clean":"2020-11-16T21:29:34.602975+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:29:34.602975+0000","last_undegraded":"2020-11-16T21:29:34.602975+0000","last_fullsized":"2020-11-16T21:29:34.602975+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607679+0000","last_change":"2020-11-16T21:29:34.604960+0000","last_active":"2020-11-16T21:29:34.607679+0000","last_peered":"2020-11-16T21:29:34.607679+0000","last_clean":"2020-11-16T21:29:34.607679+0000","last_became_active":"2020-11-16T21:29:34.604467+0000","last_became_peered":"2020-11-16T21:29:34.604467+0000","last_unstale":"2020-11-16T21:29:34.607679+0000","last_undegraded":"2020-11-16T21:29:34.607679+0000","last_fullsized":"2020-11-16T21:29:34.607679+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.606900+0000","last_change":"2020-11-16T21:29:34.606067+0000","last_active":"2020-11-16T21:29:34.606900+0000","last_peered":"2020-11-16T21:29:34.606900+0000","last_clean":"2020-11-16T21:29:34.606900+0000","last_became_active":"2020-11-16T21:29:34.605672+0000","last_became_peered":"2020-11-16T21:29:34.605672+0000","last_unstale":"2020-11-16T21:29:34.606900+0000","last_undegraded":"2020-11-16T21:29:34.606900+0000","last_fullsized":"2020-11-16T21:29:34.606900+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]}]}} 2020-11-16T21:29:46.275 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:29:46.565 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:29:46.567 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:29:46.577 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2020-11-16T21:29:46.044805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14736,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100784,"statfs":{"total":289910292480,"available":289895202816,"internally_reserved":0,"allocated":319488,"data_stored":72840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607622+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:29:34.607622+0000","last_peered":"2020-11-16T21:29:34.607622+0000","last_clean":"2020-11-16T21:29:34.607622+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:29:34.607622+0000","last_undegraded":"2020-11-16T21:29:34.607622+0000","last_fullsized":"2020-11-16T21:29:34.607622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.608057+0000","last_change":"2020-11-16T21:29:34.606262+0000","last_active":"2020-11-16T21:29:34.608057+0000","last_peered":"2020-11-16T21:29:34.608057+0000","last_clean":"2020-11-16T21:29:34.608057+0000","last_became_active":"2020-11-16T21:29:34.605864+0000","last_became_peered":"2020-11-16T21:29:34.605864+0000","last_unstale":"2020-11-16T21:29:34.608057+0000","last_undegraded":"2020-11-16T21:29:34.608057+0000","last_fullsized":"2020-11-16T21:29:34.608057+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.610836+0000","last_change":"2020-11-16T21:29:34.609971+0000","last_active":"2020-11-16T21:29:34.610836+0000","last_peered":"2020-11-16T21:29:34.610836+0000","last_clean":"2020-11-16T21:29:34.610836+0000","last_became_active":"2020-11-16T21:29:34.609635+0000","last_became_peered":"2020-11-16T21:29:34.609635+0000","last_unstale":"2020-11-16T21:29:34.610836+0000","last_undegraded":"2020-11-16T21:29:34.610836+0000","last_fullsized":"2020-11-16T21:29:34.610836+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607721+0000","last_change":"2020-11-16T21:29:34.606142+0000","last_active":"2020-11-16T21:29:34.607721+0000","last_peered":"2020-11-16T21:29:34.607721+0000","last_clean":"2020-11-16T21:29:34.607721+0000","last_became_active":"2020-11-16T21:29:34.604646+0000","last_became_peered":"2020-11-16T21:29:34.604646+0000","last_unstale":"2020-11-16T21:29:34.607721+0000","last_undegraded":"2020-11-16T21:29:34.607721+0000","last_fullsized":"2020-11-16T21:29:34.607721+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.610083+0000","last_change":"2020-11-16T21:29:34.608554+0000","last_active":"2020-11-16T21:29:34.610083+0000","last_peered":"2020-11-16T21:29:34.610083+0000","last_clean":"2020-11-16T21:29:34.610083+0000","last_became_active":"2020-11-16T21:29:34.608237+0000","last_became_peered":"2020-11-16T21:29:34.608237+0000","last_unstale":"2020-11-16T21:29:34.610083+0000","last_undegraded":"2020-11-16T21:29:34.610083+0000","last_fullsized":"2020-11-16T21:29:34.610083+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.602975+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:29:34.602975+0000","last_peered":"2020-11-16T21:29:34.602975+0000","last_clean":"2020-11-16T21:29:34.602975+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:29:34.602975+0000","last_undegraded":"2020-11-16T21:29:34.602975+0000","last_fullsized":"2020-11-16T21:29:34.602975+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.607679+0000","last_change":"2020-11-16T21:29:34.604960+0000","last_active":"2020-11-16T21:29:34.607679+0000","last_peered":"2020-11-16T21:29:34.607679+0000","last_clean":"2020-11-16T21:29:34.607679+0000","last_became_active":"2020-11-16T21:29:34.604467+0000","last_became_peered":"2020-11-16T21:29:34.604467+0000","last_unstale":"2020-11-16T21:29:34.607679+0000","last_undegraded":"2020-11-16T21:29:34.607679+0000","last_fullsized":"2020-11-16T21:29:34.607679+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"11","state":"active+clean","last_fresh":"2020-11-16T21:29:34.606900+0000","last_change":"2020-11-16T21:29:34.606067+0000","last_active":"2020-11-16T21:29:34.606900+0000","last_peered":"2020-11-16T21:29:34.606900+0000","last_clean":"2020-11-16T21:29:34.606900+0000","last_became_active":"2020-11-16T21:29:34.605672+0000","last_became_peered":"2020-11-16T21:29:34.605672+0000","last_unstale":"2020-11-16T21:29:34.606900+0000","last_undegraded":"2020-11-16T21:29:34.606900+0000","last_fullsized":"2020-11-16T21:29:34.606900+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]}]}} 2020-11-16T21:29:46.578 INFO:tasks.osd_backfill.ceph_manager:clean! 2020-11-16T21:29:46.579 INFO:tasks.osd_backfill:rados -p rbd bench 15 write -b 4096 --no-cleanup 2020-11-16T21:29:46.579 INFO:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd bench 15 write -b 4096 --no-cleanup 2020-11-16T21:29:46.619 INFO:teuthology.orchestra.run.smithi120.stdout:hints = 1 2020-11-16T21:29:46.619 INFO:teuthology.orchestra.run.smithi120.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 15 seconds or 0 objects 2020-11-16T21:29:46.620 INFO:teuthology.orchestra.run.smithi120.stdout:Object prefix: benchmark_data_smithi120_25169 2020-11-16T21:29:46.620 INFO:teuthology.orchestra.run.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T21:29:46.620 INFO:teuthology.orchestra.run.smithi120.stdout: 0 0 0 0 0 0 - 0 2020-11-16T21:29:46.663 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:46.670+0000 7f1fa395c700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6808/21984 conn(0x55b45d4e65f0 legacy=0x55b45d4e8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:46.677 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:46.685+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4ebda0 legacy=0x55b45d4ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:29:47.619 INFO:teuthology.orchestra.run.smithi120.stdout: 1 16 1925 1909 7.45647 7.45703 0.00540109 0.00831597 2020-11-16T21:29:48.482 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:48.489+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4ebda0 legacy=0x7f1f9003d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:29:48.499 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:48.507+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4f1600 legacy=0x7f1f90040d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:48.509 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:48.516+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4ebda0 legacy=0x7f1f9003d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:29:48.620 INFO:teuthology.orchestra.run.smithi120.stdout: 2 16 3750 3734 7.29243 7.12891 0.00985922 0.0085378 2020-11-16T21:29:48.697 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:48.704+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x55b45d4e65f0 legacy=0x7f1f90039ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:49.620 INFO:teuthology.orchestra.run.smithi120.stdout: 3 16 5579 5563 7.24296 7.14453 0.0134029 0.00861243 2020-11-16T21:29:49.805 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:49.813+0000 7f1fa395c700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6808/21984 conn(0x7f1f900337a0 legacy=0x7f1f90035b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:50.620 INFO:teuthology.orchestra.run.smithi120.stdout: 4 16 7353 7337 7.16452 6.92969 0.00817092 0.00870908 2020-11-16T21:29:50.771 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:50.778+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x7f1f900337a0 legacy=0x7f1f90035b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:29:51.053 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:51.060+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x55b45d4e65f0 legacy=0x7f1f9003d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:29:51.520 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:51.528+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x7f1f900337a0 legacy=0x7f1f9003e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:51.620 INFO:teuthology.orchestra.run.smithi120.stdout: 5 16 9076 9060 7.07761 6.73047 0.00913496 0.00881827 2020-11-16T21:29:52.526 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:52.534+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x55b45d4e65f0 legacy=0x7f1f9003d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:52.620 INFO:teuthology.orchestra.run.smithi120.stdout: 6 16 10695 10679 6.95197 6.32422 0.0113507 0.00897648 2020-11-16T21:29:53.620 INFO:teuthology.orchestra.run.smithi120.stdout: 7 16 12281 12265 6.8438 6.19531 0.00886963 0.00912019 2020-11-16T21:29:54.620 INFO:teuthology.orchestra.run.smithi120.stdout: 8 16 13776 13760 6.71824 5.83984 0.0126068 0.00929395 2020-11-16T21:29:54.823 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:54.831+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x7f1f900337a0 legacy=0x7f1f9003e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:55.620 INFO:teuthology.orchestra.run.smithi120.stdout: 9 16 15207 15191 6.59282 5.58984 0.0135129 0.00947199 2020-11-16T21:29:56.365 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:56.374+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4f1600 legacy=0x7f1f9003d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:56.619 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:56.627+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4e65f0 legacy=0x7f1f90036ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:29:56.620 INFO:teuthology.orchestra.run.smithi120.stdout: 10 16 16586 16570 6.47218 5.38672 0.0128866 0.00964835 2020-11-16T21:29:57.620 INFO:teuthology.orchestra.run.smithi120.stdout: 11 16 17932 17916 6.36174 5.25781 0.00838071 0.00981521 2020-11-16T21:29:58.620 INFO:teuthology.orchestra.run.smithi120.stdout: 12 16 19239 19223 6.25702 5.10547 0.0156489 0.00997926 2020-11-16T21:29:58.894 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:58.902+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4f1600 legacy=0x7f1f9003d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:58.958 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:58.966+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x7f1f900337a0 legacy=0x7f1f9003e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:59.024 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:59.033+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x55b45d4e65f0 legacy=0x7f1f90036ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:29:59.622 INFO:teuthology.orchestra.run.smithi120.stdout: 13 16 20526 20510 6.1624 5.02734 0.0112532 0.0101348 2020-11-16T21:29:59.628 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:29:59.636+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x7f1f900337a0 legacy=0x7f1f9003e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:30:00.127 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:30:00.135+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4e65f0 legacy=0x7f1f90034b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:30:00.206 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:30:00.214+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4f1600 legacy=0x7f1f9003d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:30:00.620 INFO:teuthology.orchestra.run.smithi120.stdout: 14 16 21765 21749 6.0679 4.83984 0.00977967 0.0102911 2020-11-16T21:30:01.006 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:30:01.014+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4e65f0 legacy=0x7f1f90034b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:30:01.244 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:30:01.252+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x7f1f900337a0 legacy=0x7f1f9003e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:30:01.247 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:30:01.253+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x55b45d4e65f0 legacy=0x7f1f90034b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:30:01.371 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:30:01.380+0000 7f1fa295a700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6804/21986 conn(0x55b45d4e65f0 legacy=0x7f1f90034b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:30:01.417 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:30:01.426+0000 7f1fa315b700 0 -- 172.21.15.120:0/2796187899 >> v1:172.21.15.120:6800/21985 conn(0x7f1f900337a0 legacy=0x7f1f9003e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:30:01.620 INFO:teuthology.orchestra.run.smithi120.stdout: 15 14 22970 22956 5.97768 4.71484 0.00899519 0.0104488 2020-11-16T21:30:02.621 INFO:teuthology.orchestra.run.smithi120.stdout:Total time run: 15.0109 2020-11-16T21:30:02.621 INFO:teuthology.orchestra.run.smithi120.stdout:Total writes made: 22970 2020-11-16T21:30:02.621 INFO:teuthology.orchestra.run.smithi120.stdout:Write size: 4096 2020-11-16T21:30:02.622 INFO:teuthology.orchestra.run.smithi120.stdout:Object size: 4096 2020-11-16T21:30:02.622 INFO:teuthology.orchestra.run.smithi120.stdout:Bandwidth (MB/sec): 5.97744 2020-11-16T21:30:02.622 INFO:teuthology.orchestra.run.smithi120.stdout:Stddev Bandwidth: 0.929899 2020-11-16T21:30:02.622 INFO:teuthology.orchestra.run.smithi120.stdout:Max bandwidth (MB/sec): 7.45703 2020-11-16T21:30:02.622 INFO:teuthology.orchestra.run.smithi120.stdout:Min bandwidth (MB/sec): 4.71484 2020-11-16T21:30:02.622 INFO:teuthology.orchestra.run.smithi120.stdout:Average IOPS: 1530 2020-11-16T21:30:02.623 INFO:teuthology.orchestra.run.smithi120.stdout:Stddev IOPS: 238.054 2020-11-16T21:30:02.623 INFO:teuthology.orchestra.run.smithi120.stdout:Max IOPS: 1909 2020-11-16T21:30:02.623 INFO:teuthology.orchestra.run.smithi120.stdout:Min IOPS: 1207 2020-11-16T21:30:02.623 INFO:teuthology.orchestra.run.smithi120.stdout:Average Latency(s): 0.0104494 2020-11-16T21:30:02.623 INFO:teuthology.orchestra.run.smithi120.stdout:Stddev Latency(s): 0.00354205 2020-11-16T21:30:02.623 INFO:teuthology.orchestra.run.smithi120.stdout:Max latency(s): 0.0346483 2020-11-16T21:30:02.623 INFO:teuthology.orchestra.run.smithi120.stdout:Min latency(s): 0.00224934 2020-11-16T21:30:02.627 INFO:tasks.osd_backfill:err is 0 2020-11-16T21:30:02.627 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2020-11-16T21:30:02.983 INFO:tasks.ceph.mon.a.smithi120.stderr:2020-11-16T21:30:02.991+0000 7f898acd2700 -1 mon.a@0(leader).osd e12 definitely_dead 0 2020-11-16T21:30:03.461 INFO:teuthology.orchestra.run.smithi120.stderr:marked out osd.0. 2020-11-16T21:30:03.475 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2020-11-16T21:30:03.476 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:30:03.514 INFO:tasks.ceph.osd.0:Stopped 2020-11-16T21:30:03.514 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2020-11-16T21:30:03.858 INFO:tasks.ceph.mon.a.smithi120.stderr:2020-11-16T21:30:03.867+0000 7f898acd2700 -1 mon.a@0(leader).osd e13 definitely_dead 0 2020-11-16T21:30:04.371 INFO:teuthology.orchestra.run.smithi120.stderr:marked down osd.0. 2020-11-16T21:30:04.383 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-16T21:30:04.540 INFO:teuthology.orchestra.run.smithi120.stdout:34359738378 2020-11-16T21:30:04.541 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-16T21:30:04.691 INFO:teuthology.orchestra.run.smithi120.stdout:34359738379 2020-11-16T21:30:04.691 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:30:04.979 INFO:teuthology.orchestra.run.smithi120.stdout:34359738377 2020-11-16T21:30:04.992 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738378 got 34359738377 for osd.1 2020-11-16T21:30:05.993 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:30:06.363 INFO:teuthology.orchestra.run.smithi120.stdout:34359738377 2020-11-16T21:30:06.376 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738378 got 34359738377 for osd.1 2020-11-16T21:30:07.378 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:30:07.737 INFO:teuthology.orchestra.run.smithi120.stdout:34359738378 2020-11-16T21:30:07.753 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738378 got 34359738378 for osd.1 2020-11-16T21:30:07.753 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-16T21:30:08.135 INFO:teuthology.orchestra.run.smithi120.stdout:34359738379 2020-11-16T21:30:08.152 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738379 got 34359738379 for osd.2 2020-11-16T21:30:08.152 INFO:tasks.osd_backfill.ceph_manager:waiting for recovery to complete 2020-11-16T21:30:08.152 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:08.507 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:08.510 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:08.522 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-11-16T21:30:08.048859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":171044,"kb_used_data":114900,"kb_used_omap":0,"kb_used_meta":56128,"kb_avail":188572636,"statfs":{"total":193273528320,"available":193098379264,"internally_reserved":0,"allocated":117657600,"data_stored":117461193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57475072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2830360,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000812"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"3043","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:06.048339+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:06.048339+0000","last_peered":"2020-11-16T21:30:06.048339+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:06.048339+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":274432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"2920","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:06.051648+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:06.051648+0000","last_peered":"2020-11-16T21:30:06.051648+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:06.051648+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":303104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":74}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186687488,"data_stored":186683416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70456,"kb_used_data":45488,"kb_used_omap":0,"kb_used_meta":24960,"kb_avail":94301384,"statfs":{"total":96636764160,"available":96564617216,"internally_reserved":0,"allocated":46579712,"data_stored":46476637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25559040},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738379,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100588,"kb_used_data":69412,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94271252,"statfs":{"total":96636764160,"available":96533762048,"internally_reserved":0,"allocated":71077888,"data_stored":70984556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}]}} 2020-11-16T21:30:08.524 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:08.891 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:08.893 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:08.905 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-11-16T21:30:08.048859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":171044,"kb_used_data":114900,"kb_used_omap":0,"kb_used_meta":56128,"kb_avail":188572636,"statfs":{"total":193273528320,"available":193098379264,"internally_reserved":0,"allocated":117657600,"data_stored":117461193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57475072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2830360,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000812"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"3043","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:06.048339+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:06.048339+0000","last_peered":"2020-11-16T21:30:06.048339+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:06.048339+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":274432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"2920","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:06.051648+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:06.051648+0000","last_peered":"2020-11-16T21:30:06.051648+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:06.051648+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":303104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":74}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186687488,"data_stored":186683416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70456,"kb_used_data":45488,"kb_used_omap":0,"kb_used_meta":24960,"kb_avail":94301384,"statfs":{"total":96636764160,"available":96564617216,"internally_reserved":0,"allocated":46579712,"data_stored":46476637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25559040},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738379,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100588,"kb_used_data":69412,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94271252,"statfs":{"total":96636764160,"available":96533762048,"internally_reserved":0,"allocated":71077888,"data_stored":70984556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}]}} 2020-11-16T21:30:08.906 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:09.286 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:09.289 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:09.300 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-11-16T21:30:08.048859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":171044,"kb_used_data":114900,"kb_used_omap":0,"kb_used_meta":56128,"kb_avail":188572636,"statfs":{"total":193273528320,"available":193098379264,"internally_reserved":0,"allocated":117657600,"data_stored":117461193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57475072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2830360,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000812"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"3043","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:06.048339+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:06.048339+0000","last_peered":"2020-11-16T21:30:06.048339+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:06.048339+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":274432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"2920","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:06.051648+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:06.051648+0000","last_peered":"2020-11-16T21:30:06.051648+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:06.051648+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":303104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":74}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186687488,"data_stored":186683416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70456,"kb_used_data":45488,"kb_used_omap":0,"kb_used_meta":24960,"kb_avail":94301384,"statfs":{"total":96636764160,"available":96564617216,"internally_reserved":0,"allocated":46579712,"data_stored":46476637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25559040},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738379,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100588,"kb_used_data":69412,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94271252,"statfs":{"total":96636764160,"available":96533762048,"internally_reserved":0,"allocated":71077888,"data_stored":70984556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}]}} 2020-11-16T21:30:12.305 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:12.692 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:12.695 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:12.707 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2020-11-16T21:30:12.049587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":190496,"kb_used_data":119952,"kb_used_omap":0,"kb_used_meta":70528,"kb_avail":188553184,"statfs":{"total":193273528320,"available":193078460416,"internally_reserved":0,"allocated":122830848,"data_stored":122628770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72220672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2830360,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001211"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"4389","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.050956+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:11.050956+0000","last_peered":"2020-11-16T21:30:11.050956+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:11.050956+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":3031040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"4296","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.051436+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:11.051436+0000","last_peered":"2020-11-16T21:30:11.051436+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:11.051436+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":3121152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":762}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192270336,"data_stored":192266264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79620,"kb_used_data":48060,"kb_used_omap":0,"kb_used_meta":31552,"kb_avail":94292220,"statfs":{"total":96636764160,"available":96555233280,"internally_reserved":0,"allocated":49213440,"data_stored":49110365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32309248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110876,"kb_used_data":71892,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":94260964,"statfs":{"total":96636764160,"available":96523227136,"internally_reserved":0,"allocated":73617408,"data_stored":73518405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]}]}]}} 2020-11-16T21:30:12.708 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:13.077 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:13.080 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:13.092 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2020-11-16T21:30:12.049587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":190496,"kb_used_data":119952,"kb_used_omap":0,"kb_used_meta":70528,"kb_avail":188553184,"statfs":{"total":193273528320,"available":193078460416,"internally_reserved":0,"allocated":122830848,"data_stored":122628770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72220672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2830360,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001211"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"4389","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.050956+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:11.050956+0000","last_peered":"2020-11-16T21:30:11.050956+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:11.050956+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":3031040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"4296","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.051436+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:11.051436+0000","last_peered":"2020-11-16T21:30:11.051436+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:11.051436+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":3121152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":762}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192270336,"data_stored":192266264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79620,"kb_used_data":48060,"kb_used_omap":0,"kb_used_meta":31552,"kb_avail":94292220,"statfs":{"total":96636764160,"available":96555233280,"internally_reserved":0,"allocated":49213440,"data_stored":49110365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32309248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110876,"kb_used_data":71892,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":94260964,"statfs":{"total":96636764160,"available":96523227136,"internally_reserved":0,"allocated":73617408,"data_stored":73518405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]}]}]}} 2020-11-16T21:30:16.094 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:16.459 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:16.461 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:16.474 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2020-11-16T21:30:16.050241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":190496,"kb_used_data":119952,"kb_used_omap":0,"kb_used_meta":70528,"kb_avail":188553184,"statfs":{"total":193273528320,"available":193078460416,"internally_reserved":0,"allocated":122830848,"data_stored":122628770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72220672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.676547"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"4389","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.050956+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:11.050956+0000","last_peered":"2020-11-16T21:30:11.050956+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:11.050956+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":3031040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"4296","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.051436+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:11.051436+0000","last_peered":"2020-11-16T21:30:11.051436+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:11.051436+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":3121152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":762}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192270336,"data_stored":192266264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79620,"kb_used_data":48060,"kb_used_omap":0,"kb_used_meta":31552,"kb_avail":94292220,"statfs":{"total":96636764160,"available":96555233280,"internally_reserved":0,"allocated":49213440,"data_stored":49110365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32309248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110876,"kb_used_data":71892,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":94260964,"statfs":{"total":96636764160,"available":96523227136,"internally_reserved":0,"allocated":73617408,"data_stored":73518405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]}]}]}} 2020-11-16T21:30:16.475 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:16.843 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:16.845 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:16.858 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2020-11-16T21:30:16.050241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":190496,"kb_used_data":119952,"kb_used_omap":0,"kb_used_meta":70528,"kb_avail":188553184,"statfs":{"total":193273528320,"available":193078460416,"internally_reserved":0,"allocated":122830848,"data_stored":122628770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72220672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.676547"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"4389","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.050956+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:11.050956+0000","last_peered":"2020-11-16T21:30:11.050956+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:11.050956+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":3031040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"4296","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:11.051436+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:11.051436+0000","last_peered":"2020-11-16T21:30:11.051436+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:11.051436+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":3121152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":762}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192270336,"data_stored":192266264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79620,"kb_used_data":48060,"kb_used_omap":0,"kb_used_meta":31552,"kb_avail":94292220,"statfs":{"total":96636764160,"available":96555233280,"internally_reserved":0,"allocated":49213440,"data_stored":49110365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32309248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110876,"kb_used_data":71892,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":94260964,"statfs":{"total":96636764160,"available":96523227136,"internally_reserved":0,"allocated":73617408,"data_stored":73518405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]}]}]}} 2020-11-16T21:30:19.860 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:20.254 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:20.257 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:20.269 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2020-11-16T21:30:20.050889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3260,"num_bytes_recovered":13352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":203956,"kb_used_data":126628,"kb_used_omap":0,"kb_used_meta":77312,"kb_avail":188539724,"statfs":{"total":193273528320,"available":193064677376,"internally_reserved":0,"allocated":129667072,"data_stored":129464994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79167488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3119,"num_bytes_recovered":12775424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002030"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"6135","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:16.053852+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:16.053852+0000","last_peered":"2020-11-16T21:30:16.053852+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:16.053852+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1613,"num_bytes_recovered":6606848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1613}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"6066","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:16.056400+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:16.056400+0000","last_peered":"2020-11-16T21:30:16.056400+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:16.056400+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1647,"num_bytes_recovered":6746112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3260,"num_bytes_recovered":13352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199467008,"data_stored":199462936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88440,"kb_used_data":51440,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":94283400,"statfs":{"total":96636764160,"available":96546201600,"internally_reserved":0,"allocated":52674560,"data_stored":52571485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115516,"kb_used_data":75188,"kb_used_omap":0,"kb_used_meta":40320,"kb_avail":94256324,"statfs":{"total":96636764160,"available":96518475776,"internally_reserved":0,"allocated":76992512,"data_stored":76893509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41287680},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}]}} 2020-11-16T21:30:20.270 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:20.669 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:20.670 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:20.683 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2020-11-16T21:30:20.518476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3260,"num_bytes_recovered":13352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":203956,"kb_used_data":126628,"kb_used_omap":0,"kb_used_meta":77312,"kb_avail":188539724,"statfs":{"total":193273528320,"available":193064677376,"internally_reserved":0,"allocated":129667072,"data_stored":129464994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79167488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3119,"num_bytes_recovered":12775424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.469235"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2898","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.374058+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:04.374058+0000","last_peered":"2020-11-16T21:30:04.374058+0000","last_clean":"2020-11-16T21:30:04.374058+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:04.374058+0000","last_undegraded":"2020-11-16T21:30:04.374058+0000","last_fullsized":"2020-11-16T21:30:04.374058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"6135","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:16.053852+0000","last_change":"2020-11-16T21:30:05.525653+0000","last_active":"2020-11-16T21:30:16.053852+0000","last_peered":"2020-11-16T21:30:16.053852+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:16.053852+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1613,"num_bytes_recovered":6606848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1613}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"2915","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.493441+0000","last_change":"2020-11-16T21:30:05.494685+0000","last_active":"2020-11-16T21:30:05.493441+0000","last_peered":"2020-11-16T21:30:05.493441+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:05.493441+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2945","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.533058+0000","last_change":"2020-11-16T21:30:05.534813+0000","last_active":"2020-11-16T21:30:05.533058+0000","last_peered":"2020-11-16T21:30:05.533058+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:05.533058+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2928","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.542043+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.542043+0000","last_peered":"2020-11-16T21:30:05.542043+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.542043+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2818","reported_epoch":"14","state":"active+clean","last_fresh":"2020-11-16T21:30:04.373765+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:04.373765+0000","last_peered":"2020-11-16T21:30:04.373765+0000","last_clean":"2020-11-16T21:30:04.373765+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:04.373765+0000","last_undegraded":"2020-11-16T21:30:04.373765+0000","last_fullsized":"2020-11-16T21:30:04.373765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2921","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543881+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.543881+0000","last_peered":"2020-11-16T21:30:05.543881+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.543881+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"6066","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:16.056400+0000","last_change":"2020-11-16T21:30:05.484579+0000","last_active":"2020-11-16T21:30:16.056400+0000","last_peered":"2020-11-16T21:30:16.056400+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:16.056400+0000","last_undegraded":"2020-11-16T21:30:05.389902+0000","last_fullsized":"2020-11-16T21:30:05.376024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1647,"num_bytes_recovered":6746112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3260,"num_bytes_recovered":13352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199467008,"data_stored":199462936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88440,"kb_used_data":51440,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":94283400,"statfs":{"total":96636764160,"available":96546201600,"internally_reserved":0,"allocated":52674560,"data_stored":52571485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115516,"kb_used_data":75188,"kb_used_omap":0,"kb_used_meta":40320,"kb_avail":94256324,"statfs":{"total":96636764160,"available":96518475776,"internally_reserved":0,"allocated":76992512,"data_stored":76893509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41287680},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}]}} 2020-11-16T21:30:23.684 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:24.048 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:24.051 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:24.063 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2020-11-16T21:30:24.051564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5761,"num_bytes_recovered":23592984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":197692,"kb_used_data":136684,"kb_used_omap":0,"kb_used_meta":60992,"kb_avail":188545988,"statfs":{"total":193273528320,"available":193071091712,"internally_reserved":0,"allocated":139964416,"data_stored":139762338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62455808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2501,"num_bytes_recovered":10240024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001023"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2899","reported_epoch":"15","state":"active+clean","last_fresh":"2020-11-16T21:30:05.376366+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:05.376366+0000","last_peered":"2020-11-16T21:30:05.376366+0000","last_clean":"2020-11-16T21:30:05.376366+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:05.376366+0000","last_undegraded":"2020-11-16T21:30:05.376366+0000","last_fullsized":"2020-11-16T21:30:05.376366+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8689","reported_epoch":"16","state":"active+undersized+remapped","last_fresh":"2020-11-16T21:30:20.897935+0000","last_change":"2020-11-16T21:30:20.898212+0000","last_active":"2020-11-16T21:30:20.897935+0000","last_peered":"2020-11-16T21:30:20.897935+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:20.897935+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"3117","reported_epoch":"16","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.052470+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:21.052470+0000","last_peered":"2020-11-16T21:30:21.052470+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:21.052470+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":409600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":100}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2987","reported_epoch":"16","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.055041+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:21.055041+0000","last_peered":"2020-11-16T21:30:21.055041+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:21.055041+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2929","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543142+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.543142+0000","last_peered":"2020-11-16T21:30:05.543142+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.543142+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2819","reported_epoch":"15","state":"active+clean","last_fresh":"2020-11-16T21:30:05.376246+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:05.376246+0000","last_peered":"2020-11-16T21:30:05.376246+0000","last_clean":"2020-11-16T21:30:05.376246+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:05.376246+0000","last_undegraded":"2020-11-16T21:30:05.376246+0000","last_fullsized":"2020-11-16T21:30:05.376246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2922","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.544830+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.544830+0000","last_peered":"2020-11-16T21:30:05.544830+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.544830+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8279","reported_epoch":"16","state":"peering","last_fresh":"2020-11-16T21:30:20.522184+0000","last_change":"2020-11-16T21:30:20.522597+0000","last_active":"2020-11-16T21:30:20.522184+0000","last_peered":"2020-11-16T21:30:20.244891+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:20.522184+0000","last_undegraded":"2020-11-16T21:30:20.522184+0000","last_fullsized":"2020-11-16T21:30:20.522184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5761,"num_bytes_recovered":23592984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209711104,"data_stored":209702960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83140,"kb_used_data":56380,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94288700,"statfs":{"total":96636764160,"available":96551628800,"internally_reserved":0,"allocated":57733120,"data_stored":57630045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114552,"kb_used_data":80304,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94257288,"statfs":{"total":96636764160,"available":96519462912,"internally_reserved":0,"allocated":82231296,"data_stored":82132293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}]}} 2020-11-16T21:30:24.064 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:24.433 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:24.435 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:24.449 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2020-11-16T21:30:24.051564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5761,"num_bytes_recovered":23592984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":197692,"kb_used_data":136684,"kb_used_omap":0,"kb_used_meta":60992,"kb_avail":188545988,"statfs":{"total":193273528320,"available":193071091712,"internally_reserved":0,"allocated":139964416,"data_stored":139762338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62455808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2501,"num_bytes_recovered":10240024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001023"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2899","reported_epoch":"15","state":"active+clean","last_fresh":"2020-11-16T21:30:05.376366+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:05.376366+0000","last_peered":"2020-11-16T21:30:05.376366+0000","last_clean":"2020-11-16T21:30:05.376366+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:05.376366+0000","last_undegraded":"2020-11-16T21:30:05.376366+0000","last_fullsized":"2020-11-16T21:30:05.376366+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8689","reported_epoch":"16","state":"active+undersized+remapped","last_fresh":"2020-11-16T21:30:20.897935+0000","last_change":"2020-11-16T21:30:20.898212+0000","last_active":"2020-11-16T21:30:20.897935+0000","last_peered":"2020-11-16T21:30:20.897935+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:20.897935+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"3117","reported_epoch":"16","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.052470+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:21.052470+0000","last_peered":"2020-11-16T21:30:21.052470+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:21.052470+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":409600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":100}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2987","reported_epoch":"16","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.055041+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:21.055041+0000","last_peered":"2020-11-16T21:30:21.055041+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:21.055041+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2929","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543142+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.543142+0000","last_peered":"2020-11-16T21:30:05.543142+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.543142+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2819","reported_epoch":"15","state":"active+clean","last_fresh":"2020-11-16T21:30:05.376246+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:05.376246+0000","last_peered":"2020-11-16T21:30:05.376246+0000","last_clean":"2020-11-16T21:30:05.376246+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:05.376246+0000","last_undegraded":"2020-11-16T21:30:05.376246+0000","last_fullsized":"2020-11-16T21:30:05.376246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2922","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.544830+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.544830+0000","last_peered":"2020-11-16T21:30:05.544830+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.544830+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8279","reported_epoch":"16","state":"peering","last_fresh":"2020-11-16T21:30:20.522184+0000","last_change":"2020-11-16T21:30:20.522597+0000","last_active":"2020-11-16T21:30:20.522184+0000","last_peered":"2020-11-16T21:30:20.244891+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:20.522184+0000","last_undegraded":"2020-11-16T21:30:20.522184+0000","last_fullsized":"2020-11-16T21:30:20.522184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5761,"num_bytes_recovered":23592984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209711104,"data_stored":209702960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83140,"kb_used_data":56380,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94288700,"statfs":{"total":96636764160,"available":96551628800,"internally_reserved":0,"allocated":57733120,"data_stored":57630045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114552,"kb_used_data":80304,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94257288,"statfs":{"total":96636764160,"available":96519462912,"internally_reserved":0,"allocated":82231296,"data_stored":82132293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}]}} 2020-11-16T21:30:27.450 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:27.824 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:27.827 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:27.844 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2020-11-16T21:30:26.051948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5761,"num_bytes_recovered":23592984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":197692,"kb_used_data":136684,"kb_used_omap":0,"kb_used_meta":60992,"kb_avail":188545988,"statfs":{"total":193273528320,"available":193071091712,"internally_reserved":0,"allocated":139964416,"data_stored":139762338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62455808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2501,"num_bytes_recovered":10240024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001058"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2899","reported_epoch":"15","state":"active+clean","last_fresh":"2020-11-16T21:30:05.376366+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:05.376366+0000","last_peered":"2020-11-16T21:30:05.376366+0000","last_clean":"2020-11-16T21:30:05.376366+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:05.376366+0000","last_undegraded":"2020-11-16T21:30:05.376366+0000","last_fullsized":"2020-11-16T21:30:05.376366+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8689","reported_epoch":"16","state":"active+undersized+remapped","last_fresh":"2020-11-16T21:30:20.897935+0000","last_change":"2020-11-16T21:30:20.898212+0000","last_active":"2020-11-16T21:30:20.897935+0000","last_peered":"2020-11-16T21:30:20.897935+0000","last_clean":"2020-11-16T21:30:01.634022+0000","last_became_active":"2020-11-16T21:30:05.517495+0000","last_became_peered":"2020-11-16T21:30:05.517495+0000","last_unstale":"2020-11-16T21:30:20.897935+0000","last_undegraded":"2020-11-16T21:30:05.403698+0000","last_fullsized":"2020-11-16T21:30:05.376188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"3117","reported_epoch":"16","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.052470+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:21.052470+0000","last_peered":"2020-11-16T21:30:21.052470+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:21.052470+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":409600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":100}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"2987","reported_epoch":"16","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.055041+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:21.055041+0000","last_peered":"2020-11-16T21:30:21.055041+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:21.055041+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2906,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2929","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.543142+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:05.543142+0000","last_peered":"2020-11-16T21:30:05.543142+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:05.543142+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2819","reported_epoch":"15","state":"active+clean","last_fresh":"2020-11-16T21:30:05.376246+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:05.376246+0000","last_peered":"2020-11-16T21:30:05.376246+0000","last_clean":"2020-11-16T21:30:05.376246+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:05.376246+0000","last_undegraded":"2020-11-16T21:30:05.376246+0000","last_fullsized":"2020-11-16T21:30:05.376246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2922","reported_epoch":"15","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:05.544830+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:05.544830+0000","last_peered":"2020-11-16T21:30:05.544830+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:05.544830+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8279","reported_epoch":"16","state":"peering","last_fresh":"2020-11-16T21:30:20.522184+0000","last_change":"2020-11-16T21:30:20.522597+0000","last_active":"2020-11-16T21:30:20.522184+0000","last_peered":"2020-11-16T21:30:20.244891+0000","last_clean":"2020-11-16T21:30:02.633675+0000","last_became_active":"2020-11-16T21:30:05.476375+0000","last_became_peered":"2020-11-16T21:30:05.476375+0000","last_unstale":"2020-11-16T21:30:20.522184+0000","last_undegraded":"2020-11-16T21:30:20.522184+0000","last_fullsized":"2020-11-16T21:30:20.522184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5761,"num_bytes_recovered":23592984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209711104,"data_stored":209702960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83140,"kb_used_data":56380,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94288700,"statfs":{"total":96636764160,"available":96551628800,"internally_reserved":0,"allocated":57733120,"data_stored":57630045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114552,"kb_used_data":80304,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94257288,"statfs":{"total":96636764160,"available":96519462912,"internally_reserved":0,"allocated":82231296,"data_stored":82132293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}]}} 2020-11-16T21:30:27.845 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:28.217 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:28.220 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:28.233 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2020-11-16T21:30:28.052303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7174,"num_bytes_recovered":29380632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":218548,"kb_used_data":142692,"kb_used_omap":0,"kb_used_meta":75840,"kb_avail":188525132,"statfs":{"total":193273528320,"available":193049735168,"internally_reserved":0,"allocated":146116608,"data_stored":145876234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":77660160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3914,"num_bytes_recovered":16027672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.533827"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2901","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529156+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:21.529156+0000","last_peered":"2020-11-16T21:30:21.529156+0000","last_clean":"2020-11-16T21:30:21.529156+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:21.529156+0000","last_undegraded":"2020-11-16T21:30:21.529156+0000","last_fullsized":"2020-11-16T21:30:21.529156+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8699","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.542414+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:22.542414+0000","last_peered":"2020-11-16T21:30:22.542414+0000","last_clean":"2020-11-16T21:30:22.542414+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:22.542414+0000","last_undegraded":"2020-11-16T21:30:22.542414+0000","last_fullsized":"2020-11-16T21:30:22.542414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"4635","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:26.052452+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:26.052452+0000","last_peered":"2020-11-16T21:30:26.052452+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:26.052452+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":3514368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":858}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"4299","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:26.056543+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:26.056543+0000","last_peered":"2020-11-16T21:30:26.056543+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:26.056543+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":2764800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":675}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2931","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.530341+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:21.530341+0000","last_peered":"2020-11-16T21:30:21.530341+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:21.530341+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2821","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529120+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:21.529120+0000","last_peered":"2020-11-16T21:30:21.529120+0000","last_clean":"2020-11-16T21:30:21.529120+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:21.529120+0000","last_undegraded":"2020-11-16T21:30:21.529120+0000","last_fullsized":"2020-11-16T21:30:21.529120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2924","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.532310+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:21.532310+0000","last_peered":"2020-11-16T21:30:21.532310+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:21.532310+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8288","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.538306+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:21.538306+0000","last_peered":"2020-11-16T21:30:21.538306+0000","last_clean":"2020-11-16T21:30:21.538306+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:21.538306+0000","last_undegraded":"2020-11-16T21:30:21.538306+0000","last_fullsized":"2020-11-16T21:30:21.538306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7174,"num_bytes_recovered":29380632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215498752,"data_stored":215490608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91528,"kb_used_data":59392,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94280312,"statfs":{"total":96636764160,"available":96543039488,"internally_reserved":0,"allocated":60817408,"data_stored":60697221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127020,"kb_used_data":83300,"kb_used_omap":0,"kb_used_meta":43712,"kb_avail":94244820,"statfs":{"total":96636764160,"available":96506695680,"internally_reserved":0,"allocated":85299200,"data_stored":85179013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44761088},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:30:31.235 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:31.605 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:31.608 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:31.620 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2020-11-16T21:30:30.052619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7174,"num_bytes_recovered":29380632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":218548,"kb_used_data":142692,"kb_used_omap":0,"kb_used_meta":75840,"kb_avail":188525132,"statfs":{"total":193273528320,"available":193049735168,"internally_reserved":0,"allocated":146116608,"data_stored":145876234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":77660160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":5787648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.526474"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2901","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529156+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:21.529156+0000","last_peered":"2020-11-16T21:30:21.529156+0000","last_clean":"2020-11-16T21:30:21.529156+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:21.529156+0000","last_undegraded":"2020-11-16T21:30:21.529156+0000","last_fullsized":"2020-11-16T21:30:21.529156+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8699","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.542414+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:22.542414+0000","last_peered":"2020-11-16T21:30:22.542414+0000","last_clean":"2020-11-16T21:30:22.542414+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:22.542414+0000","last_undegraded":"2020-11-16T21:30:22.542414+0000","last_fullsized":"2020-11-16T21:30:22.542414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"4635","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:26.052452+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:26.052452+0000","last_peered":"2020-11-16T21:30:26.052452+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:26.052452+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":3514368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":858}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"4299","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:26.056543+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:26.056543+0000","last_peered":"2020-11-16T21:30:26.056543+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:26.056543+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":2764800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":675}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2931","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.530341+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:21.530341+0000","last_peered":"2020-11-16T21:30:21.530341+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:21.530341+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2821","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529120+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:21.529120+0000","last_peered":"2020-11-16T21:30:21.529120+0000","last_clean":"2020-11-16T21:30:21.529120+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:21.529120+0000","last_undegraded":"2020-11-16T21:30:21.529120+0000","last_fullsized":"2020-11-16T21:30:21.529120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2924","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.532310+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:21.532310+0000","last_peered":"2020-11-16T21:30:21.532310+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:21.532310+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8288","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.538306+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:21.538306+0000","last_peered":"2020-11-16T21:30:21.538306+0000","last_clean":"2020-11-16T21:30:21.538306+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:21.538306+0000","last_undegraded":"2020-11-16T21:30:21.538306+0000","last_fullsized":"2020-11-16T21:30:21.538306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7174,"num_bytes_recovered":29380632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215498752,"data_stored":215490608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91528,"kb_used_data":59392,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94280312,"statfs":{"total":96636764160,"available":96543039488,"internally_reserved":0,"allocated":60817408,"data_stored":60697221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127020,"kb_used_data":83300,"kb_used_omap":0,"kb_used_meta":43712,"kb_avail":94244820,"statfs":{"total":96636764160,"available":96506695680,"internally_reserved":0,"allocated":85299200,"data_stored":85179013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44761088},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:30:31.621 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:31.987 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:31.990 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:32.002 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2020-11-16T21:30:30.052619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7174,"num_bytes_recovered":29380632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":218548,"kb_used_data":142692,"kb_used_omap":0,"kb_used_meta":75840,"kb_avail":188525132,"statfs":{"total":193273528320,"available":193049735168,"internally_reserved":0,"allocated":146116608,"data_stored":145876234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":77660160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":5787648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.526474"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2901","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529156+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:21.529156+0000","last_peered":"2020-11-16T21:30:21.529156+0000","last_clean":"2020-11-16T21:30:21.529156+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:21.529156+0000","last_undegraded":"2020-11-16T21:30:21.529156+0000","last_fullsized":"2020-11-16T21:30:21.529156+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8699","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.542414+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:22.542414+0000","last_peered":"2020-11-16T21:30:22.542414+0000","last_clean":"2020-11-16T21:30:22.542414+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:22.542414+0000","last_undegraded":"2020-11-16T21:30:22.542414+0000","last_fullsized":"2020-11-16T21:30:22.542414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"4635","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:26.052452+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:26.052452+0000","last_peered":"2020-11-16T21:30:26.052452+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:26.052452+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":3514368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":858}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"4299","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:26.056543+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:26.056543+0000","last_peered":"2020-11-16T21:30:26.056543+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:26.056543+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":2764800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":675}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2931","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.530341+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:21.530341+0000","last_peered":"2020-11-16T21:30:21.530341+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:21.530341+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2821","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529120+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:21.529120+0000","last_peered":"2020-11-16T21:30:21.529120+0000","last_clean":"2020-11-16T21:30:21.529120+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:21.529120+0000","last_undegraded":"2020-11-16T21:30:21.529120+0000","last_fullsized":"2020-11-16T21:30:21.529120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2924","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.532310+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:21.532310+0000","last_peered":"2020-11-16T21:30:21.532310+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:21.532310+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8288","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.538306+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:21.538306+0000","last_peered":"2020-11-16T21:30:21.538306+0000","last_clean":"2020-11-16T21:30:21.538306+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:21.538306+0000","last_undegraded":"2020-11-16T21:30:21.538306+0000","last_fullsized":"2020-11-16T21:30:21.538306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7174,"num_bytes_recovered":29380632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215498752,"data_stored":215490608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91528,"kb_used_data":59392,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94280312,"statfs":{"total":96636764160,"available":96543039488,"internally_reserved":0,"allocated":60817408,"data_stored":60697221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127020,"kb_used_data":83300,"kb_used_omap":0,"kb_used_meta":43712,"kb_avail":94244820,"statfs":{"total":96636764160,"available":96506695680,"internally_reserved":0,"allocated":85299200,"data_stored":85179013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44761088},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:30:35.005 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:35.390 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:35.392 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:35.405 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2020-11-16T21:30:34.053322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8894,"num_bytes_recovered":36425752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":223696,"kb_used_data":149248,"kb_used_omap":0,"kb_used_meta":74432,"kb_avail":188519984,"statfs":{"total":193273528320,"available":193044463616,"internally_reserved":0,"allocated":152829952,"data_stored":152589578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":76218368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3133,"num_bytes_recovered":12832768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.520358"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2901","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529156+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:21.529156+0000","last_peered":"2020-11-16T21:30:21.529156+0000","last_clean":"2020-11-16T21:30:21.529156+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:21.529156+0000","last_undegraded":"2020-11-16T21:30:21.529156+0000","last_fullsized":"2020-11-16T21:30:21.529156+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8699","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.542414+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:22.542414+0000","last_peered":"2020-11-16T21:30:22.542414+0000","last_clean":"2020-11-16T21:30:22.542414+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:22.542414+0000","last_undegraded":"2020-11-16T21:30:22.542414+0000","last_fullsized":"2020-11-16T21:30:22.542414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"6435","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:31.054680+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:31.054680+0000","last_peered":"2020-11-16T21:30:31.054680+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:31.054680+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1758,"num_bytes_recovered":7200768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"5939","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:31.058008+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:31.058008+0000","last_peered":"2020-11-16T21:30:31.058008+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:31.058008+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":6123520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2931","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.530341+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:21.530341+0000","last_peered":"2020-11-16T21:30:21.530341+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:21.530341+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2821","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529120+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:21.529120+0000","last_peered":"2020-11-16T21:30:21.529120+0000","last_clean":"2020-11-16T21:30:21.529120+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:21.529120+0000","last_undegraded":"2020-11-16T21:30:21.529120+0000","last_fullsized":"2020-11-16T21:30:21.529120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2924","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.532310+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:21.532310+0000","last_peered":"2020-11-16T21:30:21.532310+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:21.532310+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8288","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.538306+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:21.538306+0000","last_peered":"2020-11-16T21:30:21.538306+0000","last_clean":"2020-11-16T21:30:21.538306+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:21.538306+0000","last_undegraded":"2020-11-16T21:30:21.538306+0000","last_fullsized":"2020-11-16T21:30:21.538306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8894,"num_bytes_recovered":36425752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":222543872,"data_stored":222535728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100076,"kb_used_data":62564,"kb_used_omap":0,"kb_used_meta":37504,"kb_avail":94271764,"statfs":{"total":96636764160,"available":96534286336,"internally_reserved":0,"allocated":64065536,"data_stored":63945349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38404096},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123620,"kb_used_data":86684,"kb_used_omap":0,"kb_used_meta":36928,"kb_avail":94248220,"statfs":{"total":96636764160,"available":96510177280,"internally_reserved":0,"allocated":88764416,"data_stored":88644229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37814272},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]}]}} 2020-11-16T21:30:35.407 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:35.768 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:35.771 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:35.783 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2020-11-16T21:30:35.572766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8894,"num_bytes_recovered":36425752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":223696,"kb_used_data":149248,"kb_used_omap":0,"kb_used_meta":74432,"kb_avail":188519984,"statfs":{"total":193273528320,"available":193044463616,"internally_reserved":0,"allocated":152829952,"data_stored":152589578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":76218368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3133,"num_bytes_recovered":12832768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.521202"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2901","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529156+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:21.529156+0000","last_peered":"2020-11-16T21:30:21.529156+0000","last_clean":"2020-11-16T21:30:21.529156+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:21.529156+0000","last_undegraded":"2020-11-16T21:30:21.529156+0000","last_fullsized":"2020-11-16T21:30:21.529156+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8699","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.542414+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:22.542414+0000","last_peered":"2020-11-16T21:30:22.542414+0000","last_clean":"2020-11-16T21:30:22.542414+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:22.542414+0000","last_undegraded":"2020-11-16T21:30:22.542414+0000","last_fullsized":"2020-11-16T21:30:22.542414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"6435","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:31.054680+0000","last_change":"2020-11-16T21:30:20.245472+0000","last_active":"2020-11-16T21:30:31.054680+0000","last_peered":"2020-11-16T21:30:31.054680+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:31.054680+0000","last_undegraded":"2020-11-16T21:30:05.391489+0000","last_fullsized":"2020-11-16T21:30:05.376041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1758,"num_bytes_recovered":7200768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"5939","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:31.058008+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:31.058008+0000","last_peered":"2020-11-16T21:30:31.058008+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:31.058008+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":6123520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1495}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2931","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.530341+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:21.530341+0000","last_peered":"2020-11-16T21:30:21.530341+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:21.530341+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2821","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.529120+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:21.529120+0000","last_peered":"2020-11-16T21:30:21.529120+0000","last_clean":"2020-11-16T21:30:21.529120+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:21.529120+0000","last_undegraded":"2020-11-16T21:30:21.529120+0000","last_fullsized":"2020-11-16T21:30:21.529120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2924","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:21.532310+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:21.532310+0000","last_peered":"2020-11-16T21:30:21.532310+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:21.532310+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8288","reported_epoch":"17","state":"active+clean","last_fresh":"2020-11-16T21:30:21.538306+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:21.538306+0000","last_peered":"2020-11-16T21:30:21.538306+0000","last_clean":"2020-11-16T21:30:21.538306+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:21.538306+0000","last_undegraded":"2020-11-16T21:30:21.538306+0000","last_fullsized":"2020-11-16T21:30:21.538306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8894,"num_bytes_recovered":36425752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":222543872,"data_stored":222535728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100076,"kb_used_data":62564,"kb_used_omap":0,"kb_used_meta":37504,"kb_avail":94271764,"statfs":{"total":96636764160,"available":96534286336,"internally_reserved":0,"allocated":64065536,"data_stored":63945349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38404096},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123620,"kb_used_data":86684,"kb_used_omap":0,"kb_used_meta":36928,"kb_avail":94248220,"statfs":{"total":96636764160,"available":96510177280,"internally_reserved":0,"allocated":88764416,"data_stored":88644229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37814272},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]}]}} 2020-11-16T21:30:38.785 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:39.134 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:39.137 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:39.149 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2020-11-16T21:30:38.053983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11277,"num_bytes_recovered":46186520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":249296,"kb_used_data":158656,"kb_used_omap":0,"kb_used_meta":90624,"kb_avail":188494384,"statfs":{"total":193273528320,"available":193018249216,"internally_reserved":0,"allocated":162463744,"data_stored":162223370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":92798976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2383,"num_bytes_recovered":9760768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001050"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2902","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.537154+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:22.537154+0000","last_peered":"2020-11-16T21:30:22.537154+0000","last_clean":"2020-11-16T21:30:22.537154+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:22.537154+0000","last_undegraded":"2020-11-16T21:30:22.537154+0000","last_fullsized":"2020-11-16T21:30:22.537154+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8700","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.544098+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:22.544098+0000","last_peered":"2020-11-16T21:30:22.544098+0000","last_clean":"2020-11-16T21:30:22.544098+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:22.544098+0000","last_undegraded":"2020-11-16T21:30:22.544098+0000","last_fullsized":"2020-11-16T21:30:22.544098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8714","reported_epoch":"19","state":"peering","last_fresh":"2020-11-16T21:30:35.579552+0000","last_change":"2020-11-16T21:30:35.580048+0000","last_active":"2020-11-16T21:30:35.579552+0000","last_peered":"2020-11-16T21:30:35.427963+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:35.579552+0000","last_undegraded":"2020-11-16T21:30:35.579552+0000","last_fullsized":"2020-11-16T21:30:35.579552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8430","reported_epoch":"19","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.060761+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:36.060761+0000","last_peered":"2020-11-16T21:30:36.060761+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:36.060761+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":11223040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2932","reported_epoch":"18","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:22.538617+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:22.538617+0000","last_peered":"2020-11-16T21:30:22.538617+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:22.538617+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2822","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.537033+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:22.537033+0000","last_peered":"2020-11-16T21:30:22.537033+0000","last_clean":"2020-11-16T21:30:22.537033+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:22.537033+0000","last_undegraded":"2020-11-16T21:30:22.537033+0000","last_fullsized":"2020-11-16T21:30:22.537033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2925","reported_epoch":"18","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:22.538652+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:22.538652+0000","last_peered":"2020-11-16T21:30:22.538652+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:22.538652+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8289","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.535183+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:22.535183+0000","last_peered":"2020-11-16T21:30:22.535183+0000","last_clean":"2020-11-16T21:30:22.535183+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:22.535183+0000","last_undegraded":"2020-11-16T21:30:22.535183+0000","last_fullsized":"2020-11-16T21:30:22.535183+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11277,"num_bytes_recovered":46186520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":232300544,"data_stored":232292400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110480,"kb_used_data":67016,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94261360,"statfs":{"total":96636764160,"available":96523632640,"internally_reserved":0,"allocated":68624384,"data_stored":68504197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.46100000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138816,"kb_used_data":91640,"kb_used_omap":0,"kb_used_meta":47168,"kb_avail":94233024,"statfs":{"total":96636764160,"available":96494616576,"internally_reserved":0,"allocated":93839360,"data_stored":93719173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48300032},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.41699999999999998}]}]}]}} 2020-11-16T21:30:39.150 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:39.488 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:39.490 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:39.502 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2020-11-16T21:30:38.053983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11277,"num_bytes_recovered":46186520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":249296,"kb_used_data":158656,"kb_used_omap":0,"kb_used_meta":90624,"kb_avail":188494384,"statfs":{"total":193273528320,"available":193018249216,"internally_reserved":0,"allocated":162463744,"data_stored":162223370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":92798976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2383,"num_bytes_recovered":9760768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001050"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2902","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.537154+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:22.537154+0000","last_peered":"2020-11-16T21:30:22.537154+0000","last_clean":"2020-11-16T21:30:22.537154+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:22.537154+0000","last_undegraded":"2020-11-16T21:30:22.537154+0000","last_fullsized":"2020-11-16T21:30:22.537154+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8700","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.544098+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:22.544098+0000","last_peered":"2020-11-16T21:30:22.544098+0000","last_clean":"2020-11-16T21:30:22.544098+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:22.544098+0000","last_undegraded":"2020-11-16T21:30:22.544098+0000","last_fullsized":"2020-11-16T21:30:22.544098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8714","reported_epoch":"19","state":"peering","last_fresh":"2020-11-16T21:30:35.579552+0000","last_change":"2020-11-16T21:30:35.580048+0000","last_active":"2020-11-16T21:30:35.579552+0000","last_peered":"2020-11-16T21:30:35.427963+0000","last_clean":"2020-11-16T21:30:01.639653+0000","last_became_active":"2020-11-16T21:30:05.490163+0000","last_became_peered":"2020-11-16T21:30:05.490163+0000","last_unstale":"2020-11-16T21:30:35.579552+0000","last_undegraded":"2020-11-16T21:30:35.579552+0000","last_fullsized":"2020-11-16T21:30:35.579552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8430","reported_epoch":"19","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.060761+0000","last_change":"2020-11-16T21:30:20.898756+0000","last_active":"2020-11-16T21:30:36.060761+0000","last_peered":"2020-11-16T21:30:36.060761+0000","last_clean":"2020-11-16T21:30:01.636441+0000","last_became_active":"2020-11-16T21:30:05.529081+0000","last_became_peered":"2020-11-16T21:30:05.529081+0000","last_unstale":"2020-11-16T21:30:36.060761+0000","last_undegraded":"2020-11-16T21:30:05.404509+0000","last_fullsized":"2020-11-16T21:30:05.376149+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":11223040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"2932","reported_epoch":"18","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:22.538617+0000","last_change":"2020-11-16T21:30:05.543142+0000","last_active":"2020-11-16T21:30:22.538617+0000","last_peered":"2020-11-16T21:30:22.538617+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:22.538617+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2914}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2822","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.537033+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:22.537033+0000","last_peered":"2020-11-16T21:30:22.537033+0000","last_clean":"2020-11-16T21:30:22.537033+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:22.537033+0000","last_undegraded":"2020-11-16T21:30:22.537033+0000","last_fullsized":"2020-11-16T21:30:22.537033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2925","reported_epoch":"18","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:22.538652+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:22.538652+0000","last_peered":"2020-11-16T21:30:22.538652+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:22.538652+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8289","reported_epoch":"18","state":"active+clean","last_fresh":"2020-11-16T21:30:22.535183+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:22.535183+0000","last_peered":"2020-11-16T21:30:22.535183+0000","last_clean":"2020-11-16T21:30:22.535183+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:22.535183+0000","last_undegraded":"2020-11-16T21:30:22.535183+0000","last_fullsized":"2020-11-16T21:30:22.535183+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11277,"num_bytes_recovered":46186520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":232300544,"data_stored":232292400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110480,"kb_used_data":67016,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94261360,"statfs":{"total":96636764160,"available":96523632640,"internally_reserved":0,"allocated":68624384,"data_stored":68504197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.46100000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138816,"kb_used_data":91640,"kb_used_omap":0,"kb_used_meta":47168,"kb_avail":94233024,"statfs":{"total":96636764160,"available":96494616576,"internally_reserved":0,"allocated":93839360,"data_stored":93719173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48300032},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.41699999999999998}]}]}]}} 2020-11-16T21:30:42.503 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:42.834 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:42.836 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2020-11-16T21:30:42.054562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":249120,"kb_used_data":164304,"kb_used_omap":0,"kb_used_meta":84800,"kb_avail":188494560,"statfs":{"total":193273528320,"available":193018429440,"internally_reserved":0,"allocated":168247296,"data_stored":167972398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86835200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3526,"num_bytes_recovered":14442496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.481795"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2904","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587677+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:36.587677+0000","last_peered":"2020-11-16T21:30:36.587677+0000","last_clean":"2020-11-16T21:30:36.587677+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:36.587677+0000","last_undegraded":"2020-11-16T21:30:36.587677+0000","last_fullsized":"2020-11-16T21:30:36.587677+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8702","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.588071+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:36.588071+0000","last_peered":"2020-11-16T21:30:36.588071+0000","last_clean":"2020-11-16T21:30:36.588071+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:36.588071+0000","last_undegraded":"2020-11-16T21:30:36.588071+0000","last_fullsized":"2020-11-16T21:30:36.588071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8723","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.598480+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:36.598480+0000","last_peered":"2020-11-16T21:30:36.598480+0000","last_clean":"2020-11-16T21:30:36.598480+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:36.598480+0000","last_undegraded":"2020-11-16T21:30:36.598480+0000","last_fullsized":"2020-11-16T21:30:36.598480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8813","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-16T21:30:37.597499+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:37.597499+0000","last_peered":"2020-11-16T21:30:37.597499+0000","last_clean":"2020-11-16T21:30:37.597499+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:37.597499+0000","last_undegraded":"2020-11-16T21:30:37.597499+0000","last_fullsized":"2020-11-16T21:30:37.597499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"4849","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:41.056918+0000","last_change":"2020-11-16T21:30:36.380586+0000","last_active":"2020-11-16T21:30:41.056918+0000","last_peered":"2020-11-16T21:30:41.056918+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:41.056918+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":3919872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2824","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.585145+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:36.585145+0000","last_peered":"2020-11-16T21:30:36.585145+0000","last_clean":"2020-11-16T21:30:36.585145+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:36.585145+0000","last_undegraded":"2020-11-16T21:30:36.585145+0000","last_fullsized":"2020-11-16T21:30:36.585145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2927","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.586865+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:36.586865+0000","last_peered":"2020-11-16T21:30:36.586865+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:36.586865+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8291","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587565+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:36.587565+0000","last_peered":"2020-11-16T21:30:36.587565+0000","last_clean":"2020-11-16T21:30:36.587565+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:36.587565+0000","last_undegraded":"2020-11-16T21:30:36.587565+0000","last_fullsized":"2020-11-16T21:30:36.587565+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":236982272,"data_stored":236974128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109692,"kb_used_data":72244,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94262148,"statfs":{"total":96636764160,"available":96524439552,"internally_reserved":0,"allocated":73977856,"data_stored":73840407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.28199999999999997}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139428,"kb_used_data":92060,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232412,"statfs":{"total":96636764160,"available":96493989888,"internally_reserved":0,"allocated":94269440,"data_stored":94131991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.40799999999999997}]}]}]}} 2020-11-16T21:30:42.849 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:43.170 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:43.172 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:43.183 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2020-11-16T21:30:42.054562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":249120,"kb_used_data":164304,"kb_used_omap":0,"kb_used_meta":84800,"kb_avail":188494560,"statfs":{"total":193273528320,"available":193018429440,"internally_reserved":0,"allocated":168247296,"data_stored":167972398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86835200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3526,"num_bytes_recovered":14442496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.481795"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2904","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587677+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:36.587677+0000","last_peered":"2020-11-16T21:30:36.587677+0000","last_clean":"2020-11-16T21:30:36.587677+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:36.587677+0000","last_undegraded":"2020-11-16T21:30:36.587677+0000","last_fullsized":"2020-11-16T21:30:36.587677+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8702","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.588071+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:36.588071+0000","last_peered":"2020-11-16T21:30:36.588071+0000","last_clean":"2020-11-16T21:30:36.588071+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:36.588071+0000","last_undegraded":"2020-11-16T21:30:36.588071+0000","last_fullsized":"2020-11-16T21:30:36.588071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8723","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.598480+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:36.598480+0000","last_peered":"2020-11-16T21:30:36.598480+0000","last_clean":"2020-11-16T21:30:36.598480+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:36.598480+0000","last_undegraded":"2020-11-16T21:30:36.598480+0000","last_fullsized":"2020-11-16T21:30:36.598480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8813","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-16T21:30:37.597499+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:37.597499+0000","last_peered":"2020-11-16T21:30:37.597499+0000","last_clean":"2020-11-16T21:30:37.597499+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:37.597499+0000","last_undegraded":"2020-11-16T21:30:37.597499+0000","last_fullsized":"2020-11-16T21:30:37.597499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"4849","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:41.056918+0000","last_change":"2020-11-16T21:30:36.380586+0000","last_active":"2020-11-16T21:30:41.056918+0000","last_peered":"2020-11-16T21:30:41.056918+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:41.056918+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":3919872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2824","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.585145+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:36.585145+0000","last_peered":"2020-11-16T21:30:36.585145+0000","last_clean":"2020-11-16T21:30:36.585145+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:36.585145+0000","last_undegraded":"2020-11-16T21:30:36.585145+0000","last_fullsized":"2020-11-16T21:30:36.585145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2927","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.586865+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:36.586865+0000","last_peered":"2020-11-16T21:30:36.586865+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:36.586865+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8291","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587565+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:36.587565+0000","last_peered":"2020-11-16T21:30:36.587565+0000","last_clean":"2020-11-16T21:30:36.587565+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:36.587565+0000","last_undegraded":"2020-11-16T21:30:36.587565+0000","last_fullsized":"2020-11-16T21:30:36.587565+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":236982272,"data_stored":236974128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109692,"kb_used_data":72244,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94262148,"statfs":{"total":96636764160,"available":96524439552,"internally_reserved":0,"allocated":73977856,"data_stored":73840407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.28199999999999997}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139428,"kb_used_data":92060,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232412,"statfs":{"total":96636764160,"available":96493989888,"internally_reserved":0,"allocated":94269440,"data_stored":94131991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.40799999999999997}]}]}]}} 2020-11-16T21:30:46.185 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:46.538 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:46.540 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:46.551 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2020-11-16T21:30:46.055180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":249120,"kb_used_data":164304,"kb_used_omap":0,"kb_used_meta":84800,"kb_avail":188494560,"statfs":{"total":193273528320,"available":193018429440,"internally_reserved":0,"allocated":168247296,"data_stored":167972398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86835200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1143,"num_bytes_recovered":4681728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.473320"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2904","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587677+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:36.587677+0000","last_peered":"2020-11-16T21:30:36.587677+0000","last_clean":"2020-11-16T21:30:36.587677+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:36.587677+0000","last_undegraded":"2020-11-16T21:30:36.587677+0000","last_fullsized":"2020-11-16T21:30:36.587677+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8702","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.588071+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:36.588071+0000","last_peered":"2020-11-16T21:30:36.588071+0000","last_clean":"2020-11-16T21:30:36.588071+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:36.588071+0000","last_undegraded":"2020-11-16T21:30:36.588071+0000","last_fullsized":"2020-11-16T21:30:36.588071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8723","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.598480+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:36.598480+0000","last_peered":"2020-11-16T21:30:36.598480+0000","last_clean":"2020-11-16T21:30:36.598480+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:36.598480+0000","last_undegraded":"2020-11-16T21:30:36.598480+0000","last_fullsized":"2020-11-16T21:30:36.598480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8813","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-16T21:30:37.597499+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:37.597499+0000","last_peered":"2020-11-16T21:30:37.597499+0000","last_clean":"2020-11-16T21:30:37.597499+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:37.597499+0000","last_undegraded":"2020-11-16T21:30:37.597499+0000","last_fullsized":"2020-11-16T21:30:37.597499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"4849","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:41.056918+0000","last_change":"2020-11-16T21:30:36.380586+0000","last_active":"2020-11-16T21:30:41.056918+0000","last_peered":"2020-11-16T21:30:41.056918+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:41.056918+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":3919872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2824","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.585145+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:36.585145+0000","last_peered":"2020-11-16T21:30:36.585145+0000","last_clean":"2020-11-16T21:30:36.585145+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:36.585145+0000","last_undegraded":"2020-11-16T21:30:36.585145+0000","last_fullsized":"2020-11-16T21:30:36.585145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2927","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.586865+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:36.586865+0000","last_peered":"2020-11-16T21:30:36.586865+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:36.586865+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8291","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587565+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:36.587565+0000","last_peered":"2020-11-16T21:30:36.587565+0000","last_clean":"2020-11-16T21:30:36.587565+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:36.587565+0000","last_undegraded":"2020-11-16T21:30:36.587565+0000","last_fullsized":"2020-11-16T21:30:36.587565+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":236982272,"data_stored":236974128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109692,"kb_used_data":72244,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94262148,"statfs":{"total":96636764160,"available":96524439552,"internally_reserved":0,"allocated":73977856,"data_stored":73840407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.28199999999999997}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139428,"kb_used_data":92060,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232412,"statfs":{"total":96636764160,"available":96493989888,"internally_reserved":0,"allocated":94269440,"data_stored":94131991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.40799999999999997}]}]}]}} 2020-11-16T21:30:46.552 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:46.870 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:46.872 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:46.884 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2020-11-16T21:30:46.055180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":249120,"kb_used_data":164304,"kb_used_omap":0,"kb_used_meta":84800,"kb_avail":188494560,"statfs":{"total":193273528320,"available":193018429440,"internally_reserved":0,"allocated":168247296,"data_stored":167972398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86835200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1143,"num_bytes_recovered":4681728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.473320"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2904","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587677+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:36.587677+0000","last_peered":"2020-11-16T21:30:36.587677+0000","last_clean":"2020-11-16T21:30:36.587677+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:36.587677+0000","last_undegraded":"2020-11-16T21:30:36.587677+0000","last_fullsized":"2020-11-16T21:30:36.587677+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8702","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.588071+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:36.588071+0000","last_peered":"2020-11-16T21:30:36.588071+0000","last_clean":"2020-11-16T21:30:36.588071+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:36.588071+0000","last_undegraded":"2020-11-16T21:30:36.588071+0000","last_fullsized":"2020-11-16T21:30:36.588071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8723","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.598480+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:36.598480+0000","last_peered":"2020-11-16T21:30:36.598480+0000","last_clean":"2020-11-16T21:30:36.598480+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:36.598480+0000","last_undegraded":"2020-11-16T21:30:36.598480+0000","last_fullsized":"2020-11-16T21:30:36.598480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8813","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-16T21:30:37.597499+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:37.597499+0000","last_peered":"2020-11-16T21:30:37.597499+0000","last_clean":"2020-11-16T21:30:37.597499+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:37.597499+0000","last_undegraded":"2020-11-16T21:30:37.597499+0000","last_fullsized":"2020-11-16T21:30:37.597499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"4849","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:41.056918+0000","last_change":"2020-11-16T21:30:36.380586+0000","last_active":"2020-11-16T21:30:41.056918+0000","last_peered":"2020-11-16T21:30:41.056918+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:41.056918+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":3919872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2824","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.585145+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:36.585145+0000","last_peered":"2020-11-16T21:30:36.585145+0000","last_clean":"2020-11-16T21:30:36.585145+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:36.585145+0000","last_undegraded":"2020-11-16T21:30:36.585145+0000","last_fullsized":"2020-11-16T21:30:36.585145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2927","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.586865+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:36.586865+0000","last_peered":"2020-11-16T21:30:36.586865+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:36.586865+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8291","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587565+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:36.587565+0000","last_peered":"2020-11-16T21:30:36.587565+0000","last_clean":"2020-11-16T21:30:36.587565+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:36.587565+0000","last_undegraded":"2020-11-16T21:30:36.587565+0000","last_fullsized":"2020-11-16T21:30:36.587565+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12420,"num_bytes_recovered":50868248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":236982272,"data_stored":236974128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109692,"kb_used_data":72244,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94262148,"statfs":{"total":96636764160,"available":96524439552,"internally_reserved":0,"allocated":73977856,"data_stored":73840407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.28199999999999997}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139428,"kb_used_data":92060,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232412,"statfs":{"total":96636764160,"available":96493989888,"internally_reserved":0,"allocated":94269440,"data_stored":94131991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.40799999999999997}]}]}]}} 2020-11-16T21:30:49.886 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:50.225 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:50.227 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:50.238 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2020-11-16T21:30:50.055771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13522,"num_bytes_recovered":55382040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":263408,"kb_used_data":168608,"kb_used_omap":0,"kb_used_meta":94784,"kb_avail":188480272,"statfs":{"total":193273528320,"available":193003798528,"internally_reserved":0,"allocated":172654592,"data_stored":172379694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":97058816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":9195520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001508"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2904","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587677+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:36.587677+0000","last_peered":"2020-11-16T21:30:36.587677+0000","last_clean":"2020-11-16T21:30:36.587677+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:36.587677+0000","last_undegraded":"2020-11-16T21:30:36.587677+0000","last_fullsized":"2020-11-16T21:30:36.587677+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8702","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.588071+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:36.588071+0000","last_peered":"2020-11-16T21:30:36.588071+0000","last_clean":"2020-11-16T21:30:36.588071+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:36.588071+0000","last_undegraded":"2020-11-16T21:30:36.588071+0000","last_fullsized":"2020-11-16T21:30:36.588071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8723","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.598480+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:36.598480+0000","last_peered":"2020-11-16T21:30:36.598480+0000","last_clean":"2020-11-16T21:30:36.598480+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:36.598480+0000","last_undegraded":"2020-11-16T21:30:36.598480+0000","last_fullsized":"2020-11-16T21:30:36.598480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8813","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-16T21:30:37.597499+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:37.597499+0000","last_peered":"2020-11-16T21:30:37.597499+0000","last_clean":"2020-11-16T21:30:37.597499+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:37.597499+0000","last_undegraded":"2020-11-16T21:30:37.597499+0000","last_fullsized":"2020-11-16T21:30:37.597499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"7053","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:46.061336+0000","last_change":"2020-11-16T21:30:36.380586+0000","last_active":"2020-11-16T21:30:46.061336+0000","last_peered":"2020-11-16T21:30:46.061336+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:46.061336+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":8433664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2059}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2824","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.585145+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:36.585145+0000","last_peered":"2020-11-16T21:30:36.585145+0000","last_clean":"2020-11-16T21:30:36.585145+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:36.585145+0000","last_undegraded":"2020-11-16T21:30:36.585145+0000","last_fullsized":"2020-11-16T21:30:36.585145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2927","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.586865+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:36.586865+0000","last_peered":"2020-11-16T21:30:36.586865+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:36.586865+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8291","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587565+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:36.587565+0000","last_peered":"2020-11-16T21:30:36.587565+0000","last_clean":"2020-11-16T21:30:36.587565+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:36.587565+0000","last_undegraded":"2020-11-16T21:30:36.587565+0000","last_fullsized":"2020-11-16T21:30:36.587565+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13522,"num_bytes_recovered":55382040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241496064,"data_stored":241487920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123980,"kb_used_data":76548,"kb_used_omap":0,"kb_used_meta":47424,"kb_avail":94247860,"statfs":{"total":96636764160,"available":96509808640,"internally_reserved":0,"allocated":78385152,"data_stored":78247703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48562176},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.39000000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139428,"kb_used_data":92060,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232412,"statfs":{"total":96636764160,"available":96493989888,"internally_reserved":0,"allocated":94269440,"data_stored":94131991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.37}]}]}]}} 2020-11-16T21:30:50.240 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:50.582 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:50.584 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:50.596 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2020-11-16T21:30:50.380306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13522,"num_bytes_recovered":55382040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":263408,"kb_used_data":168608,"kb_used_omap":0,"kb_used_meta":94784,"kb_avail":188480272,"statfs":{"total":193273528320,"available":193003798528,"internally_reserved":0,"allocated":172654592,"data_stored":172379694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":97058816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":4513792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.325743"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2904","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587677+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:36.587677+0000","last_peered":"2020-11-16T21:30:36.587677+0000","last_clean":"2020-11-16T21:30:36.587677+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:36.587677+0000","last_undegraded":"2020-11-16T21:30:36.587677+0000","last_fullsized":"2020-11-16T21:30:36.587677+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8702","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.588071+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:36.588071+0000","last_peered":"2020-11-16T21:30:36.588071+0000","last_clean":"2020-11-16T21:30:36.588071+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:36.588071+0000","last_undegraded":"2020-11-16T21:30:36.588071+0000","last_fullsized":"2020-11-16T21:30:36.588071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8723","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.598480+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:36.598480+0000","last_peered":"2020-11-16T21:30:36.598480+0000","last_clean":"2020-11-16T21:30:36.598480+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:36.598480+0000","last_undegraded":"2020-11-16T21:30:36.598480+0000","last_fullsized":"2020-11-16T21:30:36.598480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8813","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-16T21:30:37.597499+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:37.597499+0000","last_peered":"2020-11-16T21:30:37.597499+0000","last_clean":"2020-11-16T21:30:37.597499+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:37.597499+0000","last_undegraded":"2020-11-16T21:30:37.597499+0000","last_fullsized":"2020-11-16T21:30:37.597499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"7053","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:46.061336+0000","last_change":"2020-11-16T21:30:36.380586+0000","last_active":"2020-11-16T21:30:46.061336+0000","last_peered":"2020-11-16T21:30:46.061336+0000","last_clean":"2020-11-16T21:30:01.617324+0000","last_became_active":"2020-11-16T21:30:05.538835+0000","last_became_peered":"2020-11-16T21:30:05.538835+0000","last_unstale":"2020-11-16T21:30:46.061336+0000","last_undegraded":"2020-11-16T21:30:05.406109+0000","last_fullsized":"2020-11-16T21:30:05.376125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2059,"num_bytes_recovered":8433664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2059}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2824","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.585145+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:36.585145+0000","last_peered":"2020-11-16T21:30:36.585145+0000","last_clean":"2020-11-16T21:30:36.585145+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:36.585145+0000","last_undegraded":"2020-11-16T21:30:36.585145+0000","last_fullsized":"2020-11-16T21:30:36.585145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"2927","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:36.586865+0000","last_change":"2020-11-16T21:30:05.544830+0000","last_active":"2020-11-16T21:30:36.586865+0000","last_peered":"2020-11-16T21:30:36.586865+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:36.586865+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8291","reported_epoch":"20","state":"active+clean","last_fresh":"2020-11-16T21:30:36.587565+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:36.587565+0000","last_peered":"2020-11-16T21:30:36.587565+0000","last_clean":"2020-11-16T21:30:36.587565+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:36.587565+0000","last_undegraded":"2020-11-16T21:30:36.587565+0000","last_fullsized":"2020-11-16T21:30:36.587565+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13522,"num_bytes_recovered":55382040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241496064,"data_stored":241487920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123980,"kb_used_data":76548,"kb_used_omap":0,"kb_used_meta":47424,"kb_avail":94247860,"statfs":{"total":96636764160,"available":96509808640,"internally_reserved":0,"allocated":78385152,"data_stored":78247703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48562176},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.39000000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139428,"kb_used_data":92060,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232412,"statfs":{"total":96636764160,"available":96493989888,"internally_reserved":0,"allocated":94269440,"data_stored":94131991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.37}]}]}]}} 2020-11-16T21:30:53.598 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:53.932 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:53.934 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:53.945 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2020-11-16T21:30:52.056077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14791,"num_bytes_recovered":60579864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":266728,"kb_used_data":173848,"kb_used_omap":0,"kb_used_meta":92864,"kb_avail":188476952,"statfs":{"total":193273528320,"available":193000398848,"internally_reserved":0,"allocated":178020352,"data_stored":177728084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95092736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":9711616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001160"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"3758","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:51.062413+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:30:51.062413+0000","last_peered":"2020-11-16T21:30:51.062413+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:51.062413+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14791,"num_bytes_recovered":60579864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":246689792,"data_stored":246681648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127292,"kb_used_data":81780,"kb_used_omap":0,"kb_used_meta":45504,"kb_avail":94244548,"statfs":{"total":96636764160,"available":96506417152,"internally_reserved":0,"allocated":83742720,"data_stored":83593684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46596096},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.39300000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139436,"kb_used_data":92068,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232404,"statfs":{"total":96636764160,"available":96493981696,"internally_reserved":0,"allocated":94277632,"data_stored":94134400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.313},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.39300000000000002}]}]}]}} 2020-11-16T21:30:53.946 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:54.270 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:54.272 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:54.283 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2020-11-16T21:30:54.056360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14791,"num_bytes_recovered":60579864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":266728,"kb_used_data":173848,"kb_used_omap":0,"kb_used_meta":92864,"kb_avail":188476952,"statfs":{"total":193273528320,"available":193000398848,"internally_reserved":0,"allocated":178020352,"data_stored":177728084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95092736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":9711616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001180"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"3758","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:51.062413+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:30:51.062413+0000","last_peered":"2020-11-16T21:30:51.062413+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:51.062413+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14791,"num_bytes_recovered":60579864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":246689792,"data_stored":246681648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127292,"kb_used_data":81780,"kb_used_omap":0,"kb_used_meta":45504,"kb_avail":94244548,"statfs":{"total":96636764160,"available":96506417152,"internally_reserved":0,"allocated":83742720,"data_stored":83593684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46596096},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.39300000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139436,"kb_used_data":92068,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232404,"statfs":{"total":96636764160,"available":96493981696,"internally_reserved":0,"allocated":94277632,"data_stored":94134400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.313},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.39300000000000002}]}]}]}} 2020-11-16T21:30:57.285 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:57.659 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:57.661 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:57.672 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2020-11-16T21:30:56.056614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14791,"num_bytes_recovered":60579864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":266728,"kb_used_data":173848,"kb_used_omap":0,"kb_used_meta":92864,"kb_avail":188476952,"statfs":{"total":193273528320,"available":193000398848,"internally_reserved":0,"allocated":178020352,"data_stored":177728084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95092736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1269,"num_bytes_recovered":5197824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001144"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"3758","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:51.062413+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:30:51.062413+0000","last_peered":"2020-11-16T21:30:51.062413+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:51.062413+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14791,"num_bytes_recovered":60579864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":246689792,"data_stored":246681648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127292,"kb_used_data":81780,"kb_used_omap":0,"kb_used_meta":45504,"kb_avail":94244548,"statfs":{"total":96636764160,"available":96506417152,"internally_reserved":0,"allocated":83742720,"data_stored":83593684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46596096},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.39300000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139436,"kb_used_data":92068,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232404,"statfs":{"total":96636764160,"available":96493981696,"internally_reserved":0,"allocated":94277632,"data_stored":94134400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.313},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.39300000000000002}]}]}]}} 2020-11-16T21:30:57.673 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:30:58.056 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:30:58.058 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:30:58.071 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2020-11-16T21:30:58.056880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15690,"num_bytes_recovered":64262168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":279784,"kb_used_data":177304,"kb_used_omap":0,"kb_used_meta":102464,"kb_avail":188463896,"statfs":{"total":193273528320,"available":192987029504,"internally_reserved":0,"allocated":181559296,"data_stored":181261224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104923136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2168,"num_bytes_recovered":8880128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.680964"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"5556","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:56.064563+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:30:56.064563+0000","last_peered":"2020-11-16T21:30:56.064563+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:56.064563+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":5378048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1313}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15690,"num_bytes_recovered":64262168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250372096,"data_stored":250363952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":140340,"kb_used_data":85228,"kb_used_omap":0,"kb_used_meta":55104,"kb_avail":94231500,"statfs":{"total":96636764160,"available":96493056000,"internally_reserved":0,"allocated":87273472,"data_stored":87124436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56426496},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.41099999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139444,"kb_used_data":92076,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232396,"statfs":{"total":96636764160,"available":96493973504,"internally_reserved":0,"allocated":94285824,"data_stored":94136788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.36399999999999999}]}]}]}} 2020-11-16T21:31:01.073 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:01.420 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:01.423 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:01.434 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2020-11-16T21:31:00.057159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15690,"num_bytes_recovered":64262168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":279784,"kb_used_data":177304,"kb_used_omap":0,"kb_used_meta":102464,"kb_avail":188463896,"statfs":{"total":193273528320,"available":192987029504,"internally_reserved":0,"allocated":181559296,"data_stored":181261224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104923136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2168,"num_bytes_recovered":8880128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001387"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"5556","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:56.064563+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:30:56.064563+0000","last_peered":"2020-11-16T21:30:56.064563+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:56.064563+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":5378048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1313}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15690,"num_bytes_recovered":64262168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250372096,"data_stored":250363952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":140340,"kb_used_data":85228,"kb_used_omap":0,"kb_used_meta":55104,"kb_avail":94231500,"statfs":{"total":96636764160,"available":96493056000,"internally_reserved":0,"allocated":87273472,"data_stored":87124436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56426496},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.41099999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139444,"kb_used_data":92076,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232396,"statfs":{"total":96636764160,"available":96493973504,"internally_reserved":0,"allocated":94285824,"data_stored":94136788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.36399999999999999}]}]}]}} 2020-11-16T21:31:01.435 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:01.766 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:01.777 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2020-11-16T21:31:00.057159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15690,"num_bytes_recovered":64262168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":279784,"kb_used_data":177304,"kb_used_omap":0,"kb_used_meta":102464,"kb_avail":188463896,"statfs":{"total":193273528320,"available":192987029504,"internally_reserved":0,"allocated":181559296,"data_stored":181261224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104923136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2168,"num_bytes_recovered":8880128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001387"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"5556","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:30:56.064563+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:30:56.064563+0000","last_peered":"2020-11-16T21:30:56.064563+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:30:56.064563+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":5378048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1313}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15690,"num_bytes_recovered":64262168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250372096,"data_stored":250363952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":140340,"kb_used_data":85228,"kb_used_omap":0,"kb_used_meta":55104,"kb_avail":94231500,"statfs":{"total":96636764160,"available":96493056000,"internally_reserved":0,"allocated":87273472,"data_stored":87124436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56426496},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.41099999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139444,"kb_used_data":92076,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232396,"statfs":{"total":96636764160,"available":96493973504,"internally_reserved":0,"allocated":94285824,"data_stored":94136788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.36399999999999999}]}]}]}} 2020-11-16T21:31:04.779 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:05.079 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:05.089 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2020-11-16T21:31:04.057850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16915,"num_bytes_recovered":69279768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":278076,"kb_used_data":181868,"kb_used_omap":0,"kb_used_meta":96192,"kb_avail":188465604,"statfs":{"total":193273528320,"available":192988778496,"internally_reserved":0,"allocated":186232832,"data_stored":185934760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98500608},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":8699904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001236"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"8006","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:01.067108+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:31:01.067108+0000","last_peered":"2020-11-16T21:31:01.067108+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:31:01.067108+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2538,"num_bytes_recovered":10395648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2538}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16915,"num_bytes_recovered":69279768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":255385600,"data_stored":255377456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138632,"kb_used_data":89792,"kb_used_omap":0,"kb_used_meta":48832,"kb_avail":94233208,"statfs":{"total":96636764160,"available":96494804992,"internally_reserved":0,"allocated":91947008,"data_stored":91797972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50003968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.29899999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139444,"kb_used_data":92076,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232396,"statfs":{"total":96636764160,"available":96493973504,"internally_reserved":0,"allocated":94285824,"data_stored":94136788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.373},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.34200000000000003}]}]}]}} 2020-11-16T21:31:05.091 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:05.388 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2020-11-16T21:31:04.057850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16915,"num_bytes_recovered":69279768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":278076,"kb_used_data":181868,"kb_used_omap":0,"kb_used_meta":96192,"kb_avail":188465604,"statfs":{"total":193273528320,"available":192988778496,"internally_reserved":0,"allocated":186232832,"data_stored":185934760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98500608},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":8699904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001236"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2906","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378227+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:30:49.378227+0000","last_peered":"2020-11-16T21:30:49.378227+0000","last_clean":"2020-11-16T21:30:49.378227+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:30:49.378227+0000","last_undegraded":"2020-11-16T21:30:49.378227+0000","last_fullsized":"2020-11-16T21:30:49.378227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8704","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378245+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:30:49.378245+0000","last_peered":"2020-11-16T21:30:49.378245+0000","last_clean":"2020-11-16T21:30:49.378245+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:30:49.378245+0000","last_undegraded":"2020-11-16T21:30:49.378245+0000","last_fullsized":"2020-11-16T21:30:49.378245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8725","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381155+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:30:49.381155+0000","last_peered":"2020-11-16T21:30:49.381155+0000","last_clean":"2020-11-16T21:30:49.381155+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:30:49.381155+0000","last_undegraded":"2020-11-16T21:30:49.381155+0000","last_fullsized":"2020-11-16T21:30:49.381155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8815","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378426+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:30:49.378426+0000","last_peered":"2020-11-16T21:30:49.378426+0000","last_clean":"2020-11-16T21:30:49.378426+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:30:49.378426+0000","last_undegraded":"2020-11-16T21:30:49.378426+0000","last_fullsized":"2020-11-16T21:30:49.378426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8774","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-16T21:30:50.388555+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:30:50.388555+0000","last_peered":"2020-11-16T21:30:50.388555+0000","last_clean":"2020-11-16T21:30:50.388555+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:30:50.388555+0000","last_undegraded":"2020-11-16T21:30:50.388555+0000","last_fullsized":"2020-11-16T21:30:50.388555+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2826","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.378186+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:30:49.378186+0000","last_peered":"2020-11-16T21:30:49.378186+0000","last_clean":"2020-11-16T21:30:49.378186+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:30:49.378186+0000","last_undegraded":"2020-11-16T21:30:49.378186+0000","last_fullsized":"2020-11-16T21:30:49.378186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"8006","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:01.067108+0000","last_change":"2020-11-16T21:30:48.837084+0000","last_active":"2020-11-16T21:31:01.067108+0000","last_peered":"2020-11-16T21:31:01.067108+0000","last_clean":"2020-11-16T21:30:01.625488+0000","last_became_active":"2020-11-16T21:30:05.541384+0000","last_became_peered":"2020-11-16T21:30:05.541384+0000","last_unstale":"2020-11-16T21:31:01.067108+0000","last_undegraded":"2020-11-16T21:30:05.406112+0000","last_fullsized":"2020-11-16T21:30:05.376296+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2538,"num_bytes_recovered":10395648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2538}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8293","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-16T21:30:49.381418+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:30:49.381418+0000","last_peered":"2020-11-16T21:30:49.381418+0000","last_clean":"2020-11-16T21:30:49.381418+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:30:49.381418+0000","last_undegraded":"2020-11-16T21:30:49.381418+0000","last_fullsized":"2020-11-16T21:30:49.381418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16915,"num_bytes_recovered":69279768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":255385600,"data_stored":255377456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138632,"kb_used_data":89792,"kb_used_omap":0,"kb_used_meta":48832,"kb_avail":94233208,"statfs":{"total":96636764160,"available":96494804992,"internally_reserved":0,"allocated":91947008,"data_stored":91797972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50003968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.29899999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139444,"kb_used_data":92076,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232396,"statfs":{"total":96636764160,"available":96493973504,"internally_reserved":0,"allocated":94285824,"data_stored":94136788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.373},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.34200000000000003}]}]}]}} 2020-11-16T21:31:08.390 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:08.691 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:08.693 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:08.703 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2020-11-16T21:31:08.058670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17279,"num_bytes_recovered":70770712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":285448,"kb_used_data":184184,"kb_used_omap":0,"kb_used_meta":101248,"kb_avail":188458232,"statfs":{"total":193273528320,"available":192981229568,"internally_reserved":0,"allocated":188604416,"data_stored":188283070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":103677952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":6508544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001510"},"pg_stats":[{"pgid":"1.7","version":"12'2886","reported_seq":"2908","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-16T21:31:02.403824+0000","last_change":"2020-11-16T21:29:34.600872+0000","last_active":"2020-11-16T21:31:02.403824+0000","last_peered":"2020-11-16T21:31:02.403824+0000","last_clean":"2020-11-16T21:31:02.403824+0000","last_became_active":"2020-11-16T21:29:34.600488+0000","last_became_peered":"2020-11-16T21:29:34.600488+0000","last_unstale":"2020-11-16T21:31:02.403824+0000","last_undegraded":"2020-11-16T21:31:02.403824+0000","last_fullsized":"2020-11-16T21:31:02.403824+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2886,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"12'2889","reported_seq":"8706","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-16T21:31:02.404071+0000","last_change":"2020-11-16T21:30:22.542414+0000","last_active":"2020-11-16T21:31:02.404071+0000","last_peered":"2020-11-16T21:31:02.404071+0000","last_clean":"2020-11-16T21:31:02.404071+0000","last_became_active":"2020-11-16T21:30:22.539789+0000","last_became_peered":"2020-11-16T21:30:22.539789+0000","last_unstale":"2020-11-16T21:31:02.404071+0000","last_undegraded":"2020-11-16T21:31:02.404071+0000","last_fullsized":"2020-11-16T21:31:02.404071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2889,"ondisk_log_size":2889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11833344,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2889,"num_write_kb":11556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"12'2896","reported_seq":"8727","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-16T21:31:02.406488+0000","last_change":"2020-11-16T21:30:36.595280+0000","last_active":"2020-11-16T21:31:02.406488+0000","last_peered":"2020-11-16T21:31:02.406488+0000","last_clean":"2020-11-16T21:31:02.406488+0000","last_became_active":"2020-11-16T21:30:36.592778+0000","last_became_peered":"2020-11-16T21:30:36.592778+0000","last_unstale":"2020-11-16T21:31:02.406488+0000","last_undegraded":"2020-11-16T21:31:02.406488+0000","last_fullsized":"2020-11-16T21:31:02.406488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2896,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11862016,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'2926","reported_seq":"8817","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-16T21:31:02.404133+0000","last_change":"2020-11-16T21:30:37.597499+0000","last_active":"2020-11-16T21:31:02.404133+0000","last_peered":"2020-11-16T21:31:02.404133+0000","last_clean":"2020-11-16T21:31:02.404133+0000","last_became_active":"2020-11-16T21:30:37.594713+0000","last_became_peered":"2020-11-16T21:30:37.594713+0000","last_unstale":"2020-11-16T21:31:02.404133+0000","last_undegraded":"2020-11-16T21:31:02.404133+0000","last_fullsized":"2020-11-16T21:31:02.404133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2926,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":11704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2914","reported_seq":"8776","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-16T21:31:02.403892+0000","last_change":"2020-11-16T21:30:50.388555+0000","last_active":"2020-11-16T21:31:02.403892+0000","last_peered":"2020-11-16T21:31:02.403892+0000","last_clean":"2020-11-16T21:31:02.403892+0000","last_became_active":"2020-11-16T21:30:50.385908+0000","last_became_peered":"2020-11-16T21:30:50.385908+0000","last_unstale":"2020-11-16T21:31:02.403892+0000","last_undegraded":"2020-11-16T21:31:02.403892+0000","last_fullsized":"2020-11-16T21:31:02.403892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2914,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11935744,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":11656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'2806","reported_seq":"2828","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-16T21:31:02.403887+0000","last_change":"2020-11-16T21:29:34.599170+0000","last_active":"2020-11-16T21:31:02.403887+0000","last_peered":"2020-11-16T21:31:02.403887+0000","last_clean":"2020-11-16T21:31:02.403887+0000","last_became_active":"2020-11-16T21:29:34.598744+0000","last_became_peered":"2020-11-16T21:29:34.598744+0000","last_unstale":"2020-11-16T21:31:02.403887+0000","last_undegraded":"2020-11-16T21:31:02.403887+0000","last_fullsized":"2020-11-16T21:31:02.403887+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11493376,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":11224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"12'2902","reported_seq":"8745","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-16T21:31:03.412345+0000","last_change":"2020-11-16T21:31:03.412345+0000","last_active":"2020-11-16T21:31:03.412345+0000","last_peered":"2020-11-16T21:31:03.412345+0000","last_clean":"2020-11-16T21:31:03.412345+0000","last_became_active":"2020-11-16T21:31:03.409402+0000","last_became_peered":"2020-11-16T21:31:03.409402+0000","last_unstale":"2020-11-16T21:31:03.412345+0000","last_undegraded":"2020-11-16T21:31:03.412345+0000","last_fullsized":"2020-11-16T21:31:03.412345+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2902,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11886592,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"12'2752","reported_seq":"8295","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-16T21:31:02.406589+0000","last_change":"2020-11-16T21:30:21.537010+0000","last_active":"2020-11-16T21:31:02.406589+0000","last_peered":"2020-11-16T21:31:02.406589+0000","last_clean":"2020-11-16T21:31:02.406589+0000","last_became_active":"2020-11-16T21:30:21.533933+0000","last_became_peered":"2020-11-16T21:30:21.533933+0000","last_unstale":"2020-11-16T21:31:02.406589+0000","last_undegraded":"2020-11-16T21:31:02.406589+0000","last_fullsized":"2020-11-16T21:31:02.406589+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":2752,"ondisk_log_size":2752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268120,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":11005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":94085144,"num_objects":22971,"num_object_clones":0,"num_object_copies":45942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22971,"num_write_kb":91881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17279,"num_bytes_recovered":70770712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":256884736,"data_stored":256876592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22971,"ondisk_log_size":22971,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145988,"kb_used_data":92092,"kb_used_omap":0,"kb_used_meta":53888,"kb_avail":94225852,"statfs":{"total":96636764160,"available":96487272448,"internally_reserved":0,"allocated":94302208,"data_stored":94141535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55181312},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.308},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.42199999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139460,"kb_used_data":92092,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":94232380,"statfs":{"total":96636764160,"available":96493957120,"internally_reserved":0,"allocated":94302208,"data_stored":94141535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":2,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.47999999999999998}]}]}]}} 2020-11-16T21:31:08.704 INFO:tasks.osd_backfill.ceph_manager:recovered! 2020-11-16T21:31:08.704 INFO:tasks.osd_backfill:rados -p rbd bench 30 write -b 4096 --no-cleanup 2020-11-16T21:31:08.705 INFO:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd bench 30 write -b 4096 --no-cleanup 2020-11-16T21:31:08.745 INFO:teuthology.orchestra.run.smithi120.stdout:hints = 1 2020-11-16T21:31:08.745 INFO:teuthology.orchestra.run.smithi120.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2020-11-16T21:31:08.745 INFO:teuthology.orchestra.run.smithi120.stdout:Object prefix: benchmark_data_smithi120_26989 2020-11-16T21:31:08.746 INFO:teuthology.orchestra.run.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T21:31:08.746 INFO:teuthology.orchestra.run.smithi120.stdout: 0 0 0 0 0 0 - 0 2020-11-16T21:31:08.828 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:08.835+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928ce4a0 legacy=0x55d3928d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:08.847 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:08.854+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x7f6e100336a0 legacy=0x7f6e10035a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:09.745 INFO:teuthology.orchestra.run.smithi120.stdout: 1 16 1194 1178 4.60131 4.60156 0.0153134 0.0134749 2020-11-16T21:31:10.745 INFO:teuthology.orchestra.run.smithi120.stdout: 2 16 2340 2324 4.53877 4.47656 0.0141599 0.0137143 2020-11-16T21:31:11.745 INFO:teuthology.orchestra.run.smithi120.stdout: 3 16 3365 3349 4.36038 4.00391 0.0106548 0.0142192 2020-11-16T21:31:11.869 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:11.876+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:11.893 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:11.901+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928c96f0 legacy=0x7f6e10037d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:11.907 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:11.915+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x7f6e100336a0 legacy=0x7f6e10038fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:12.237 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:12.244+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928c96f0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:12.745 INFO:teuthology.orchestra.run.smithi120.stdout: 4 16 4441 4425 4.32098 4.20312 0.0187715 0.0144197 2020-11-16T21:31:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: 5 16 5521 5505 4.30047 4.21875 0.0122 0.0145097 2020-11-16T21:31:14.103 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:14.111+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10036aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: 6 16 6591 6575 4.28029 4.17969 0.0131464 0.0145747 2020-11-16T21:31:15.710 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:15.717+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928ce4a0 legacy=0x7f6e10036a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:15.746 INFO:teuthology.orchestra.run.smithi120.stdout: 7 16 7634 7618 4.2508 4.07422 0.0131903 0.0146724 2020-11-16T21:31:16.173 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:16.181+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:16.746 INFO:teuthology.orchestra.run.smithi120.stdout: 8 16 8669 8653 4.22478 4.04297 0.0182404 0.0147794 2020-11-16T21:31:16.950 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:16.957+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:17.748 INFO:teuthology.orchestra.run.smithi120.stdout: 9 16 9665 9649 4.18762 3.89062 0.019574 0.0149065 2020-11-16T21:31:18.602 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:18.608+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928ce4a0 legacy=0x7f6e10036a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:18.746 INFO:teuthology.orchestra.run.smithi120.stdout: 10 16 10653 10637 4.15477 3.85938 0.0131487 0.0150202 2020-11-16T21:31:19.746 INFO:teuthology.orchestra.run.smithi120.stdout: 11 16 11615 11599 4.11867 3.75781 0.0148694 0.0151601 2020-11-16T21:31:20.746 INFO:teuthology.orchestra.run.smithi120.stdout: 12 16 12566 12550 4.08499 3.71484 0.0121215 0.0152869 2020-11-16T21:31:21.746 INFO:teuthology.orchestra.run.smithi120.stdout: 13 16 13491 13475 4.04868 3.61328 0.0153652 0.0154184 2020-11-16T21:31:22.341 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:22.349+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:22.746 INFO:teuthology.orchestra.run.smithi120.stdout: 14 16 14403 14387 4.01394 3.5625 0.0273181 0.0155594 2020-11-16T21:31:23.708 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell osd.1 injectargs --objectstore-blackhole=true 2020-11-16T21:31:23.746 INFO:teuthology.orchestra.run.smithi120.stdout: 15 16 15300 15284 3.97992 3.50391 0.0169467 0.0156952 2020-11-16T21:31:24.011 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.019+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.012 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.020+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.013 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.021+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.013 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.021+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.014 INFO:teuthology.orchestra.run.smithi120.stdout:{} 2020-11-16T21:31:24.017 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.023+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.024+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.025+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.026+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.026+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.019 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.027+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.020 INFO:teuthology.orchestra.run.smithi120.stderr:objectstore_blackhole = 'true' (not observed, change may require restart) 2020-11-16T21:31:24.021 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.028+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.021 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.029+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.022 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.029+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.022 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.030+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.022 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.030+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.031+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.031+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.032+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.026 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.033+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.027 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.033+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.027 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.033+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.027 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.034+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.027 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.035+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.027 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.035+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.028 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.036+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.028 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.036+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.038+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.038+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.039+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.034 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.039+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.034 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.041+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.042+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.042+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.043+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.036 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.044+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.036 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.045+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.037 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.046+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.047+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.049+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.049+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.050+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.050+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.044 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.050+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.044 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.051+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.044 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.052+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.045 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.052+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.045 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.052+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.045 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.052+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.046 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.054+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.046 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.054+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.046 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.055+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.047 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.056+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.048 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.056+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.048 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.057+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.049 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.057+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.053 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.058+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.053 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.059+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.053 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.060+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.054 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.060+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.054 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.060+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.054 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.060+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.054 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.060+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.055 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.061+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.055 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.063+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.058 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.064+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.058 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.064+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.058 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.065+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.058 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.065+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.059 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.065+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.059 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.065+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.066+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.066+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.066+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.066+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.067+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.068+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.061 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.068+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.061 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.068+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.061 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.069+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.061 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.069+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.062 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.070+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.062 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.071+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.063 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.071+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.064 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.072+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.064 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.072+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.066 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.073+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.067 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.073+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.067 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.073+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.067 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.074+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.068 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.074+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.068 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.075+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.068 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.077+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.069 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.077+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.069 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.078+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.070 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.078+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.070 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.078+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.071 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.080+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.071 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.080+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.073 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.081+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.073 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.081+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.073 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.081+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.074 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.082+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.075 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.083+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.075 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.084+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.076 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.084+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.076 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.085+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.077 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.085+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.079 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.087+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.079 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.087+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.079 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.088+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.079 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.088+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.080 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.088+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.080 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.088+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.082 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.091+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.083 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.091+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.086 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.092+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.086 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.093+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.087 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.093+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.087 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.093+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.087 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.094+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.088 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.095+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.088 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.095+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.088 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.095+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.088 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.096+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.089 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.096+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.089 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.097+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.089 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.097+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.089 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.098+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.092 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.098+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.092 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.098+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.093 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.099+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.093 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.099+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.093 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.100+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.093 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.100+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.094 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.101+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.094 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.101+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.095 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.101+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.095 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.102+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.095 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.103+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.095 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.103+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.095 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.103+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.096 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.104+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.096 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.104+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.097 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.105+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.097 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.106+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.098 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.106+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.098 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.107+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.102 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.107+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.103 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.108+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.103 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.108+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.104 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.110+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.104 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.111+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.105 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.111+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.105 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.112+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.106 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.112+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.106 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.113+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.106 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.113+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.106 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.114+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.106 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.115+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.107 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.116+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.109 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.117+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.109 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.118+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.110 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.118+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.110 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.118+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.111 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.119+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.111 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.119+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.112 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.119+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.113 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.122+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.114 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.123+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.114 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.123+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.115 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.123+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.117 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.123+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.117 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.124+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.118 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.124+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.118 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.126+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.119 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.126+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.119 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.126+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.119 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.127+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.120 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.128+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.120 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.128+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.122 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.129+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.123 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.130+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.123 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.130+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.124 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.130+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.124 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.131+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.124 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.131+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.124 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.131+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.125 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.133+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.125 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.133+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.125 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.133+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.125 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.134+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.126 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.134+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.127 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.135+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.127 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.135+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.128 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.136+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.128 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.136+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.128 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.136+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.129 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.137+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.129 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.138+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.129 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.138+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.130 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.138+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.130 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.138+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.132 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.140+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.133 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.140+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.133 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.141+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.133 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.141+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.134 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.142+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.134 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.142+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.134 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.143+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.135 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.143+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.135 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.143+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.136 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.144+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.136 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.144+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.136 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.144+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.136 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.145+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.137 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.146+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.137 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.146+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.138 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.146+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.138 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.147+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.139 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.148+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.139 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.148+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.148+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.148+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.149+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.141 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.149+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.141 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.150+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.151+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.143 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.152+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.143 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.152+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.143 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.152+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.144 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.153+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.144 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.153+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.144 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.153+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.145 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.154+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.145 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.154+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.145 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.154+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.147 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.155+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.147 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.155+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.148 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.157+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.148 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.157+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.149 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.157+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.149 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.158+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.150 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.158+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.150 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.158+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.150 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.159+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.151 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.160+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.152 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.160+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.152 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.160+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.153 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.161+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.153 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.161+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.153 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.162+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.154 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.162+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.154 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.163+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.154 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.163+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.155 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.163+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.158 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.165+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.158 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.165+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.158 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.166+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.159 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.166+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.159 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.166+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.159 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.167+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.159 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.167+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.160 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.167+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.160 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.169+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.160 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.169+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.161 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.169+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.161 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.169+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.163 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.171+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.164 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.173+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.164 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.173+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.165 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.173+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.165 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.174+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.166 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.174+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.166 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.174+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.166 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.174+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.175+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.175+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.175+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.176+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.168 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.177+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.168 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.177+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.171 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.178+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.172 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.178+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.172 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.178+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.180+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.180+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.180+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.181+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.182+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.174 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.182+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.174 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.182+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.177 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.184+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.177 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.185+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.178 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.185+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.178 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.185+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.178 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.185+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.179 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.186+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.180 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.188+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.180 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.188+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.181 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.189+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.181 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.190+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.183 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.192+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.184 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.192+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.185 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.193+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.186 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.194+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.186 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.194+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.186 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.194+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.187 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.195+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.187 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.196+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.187 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.196+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.188 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.196+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.190 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.197+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.190 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.197+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.190 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.197+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.191 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.198+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.191 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.200+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.192 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.200+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.192 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.201+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.193 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.202+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.194 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.202+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.194 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.202+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.194 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.203+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.195 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.203+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.198 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.204+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.201 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.205+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.201 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.205+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.201 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.205+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.202 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.207+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.202 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.207+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.202 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.207+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.203 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.208+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.203 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.208+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.203 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.208+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.205 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.208+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.206 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.208+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.206 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.210+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.206 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.210+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.206 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.210+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.207 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.210+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.207 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.211+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.207 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.211+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.208 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.213+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.208 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.214+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.208 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.214+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.209 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.214+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.209 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.214+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.210 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.215+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.210 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.216+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.210 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.217+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.210 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.217+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.210 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.217+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.210 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.218+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.211 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.219+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.211 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.220+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.212 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.220+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.212 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.220+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.213 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.222+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.213 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.222+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.214 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.222+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.214 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.223+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.214 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.223+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.216 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.224+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.219 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.225+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.219 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.227+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.219 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.227+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.219 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.227+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.220 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.228+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.220 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.228+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.220 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.228+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.222 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.229+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.225 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.229+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.225 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.230+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.233+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.233+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.233+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.233+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.228 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.235+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.228 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.235+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.228 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.236+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.228 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.236+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.230 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.237+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.238+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.238+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.238+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.239+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.239+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.232 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.240+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.232 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.240+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.232 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.241+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.232 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.241+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.233 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.241+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.233 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.242+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.234 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.243+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.234 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.243+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.238 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.244+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.238 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.244+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.239 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.245+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.239 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.245+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.239 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.247+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.240 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.248+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.240 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.248+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.240 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.249+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.241 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.250+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.241 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.250+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.241 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.250+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.242 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.251+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.242 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.251+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.243 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.251+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.243 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.252+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.244 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.253+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.245 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.253+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.245 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.254+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.246 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.254+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.246 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.255+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.247 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.255+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.248 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.257+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.248 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.257+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.249 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.258+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.249 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.258+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.250 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.258+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.250 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.259+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.251 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.259+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.251 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.260+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.252 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.260+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.252 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.261+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.252 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.261+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.253 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.262+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.253 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.262+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.254 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.262+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.254 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.263+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.258 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.264+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.258 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.264+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.259 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.265+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.259 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.266+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.259 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.266+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.260 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.267+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.260 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.267+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.260 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.267+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.261 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.267+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.261 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.268+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.261 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.268+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.261 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.269+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.261 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.269+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.264 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.270+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.267 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.270+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.267 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.273+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.269 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.274+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.269 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.274+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.269 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.274+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.270 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.274+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.270 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.275+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.270 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.277+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.270 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.277+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.271 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.277+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.271 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.278+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.271 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.278+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.272 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.279+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.272 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.280+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.272 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.280+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.272 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.280+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.273 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.280+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.273 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.281+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.274 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.283+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.283+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.284+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.284+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.285+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.279 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.285+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.281 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.285+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.281 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.287+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.282 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.287+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.282 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.287+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.283 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.288+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.283 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.288+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.283 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.289+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.283 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.289+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.290+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.290+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.290+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.290+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.291+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.291+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.292+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.286 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.292+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.286 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.293+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.286 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.293+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.286 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.293+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.287 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.294+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.289 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.294+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.289 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.295+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.289 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.295+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.290 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.296+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.290 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.296+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.290 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.297+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.290 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.297+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.295 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.297+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.295 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.298+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.296 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.299+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.296 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.299+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.299 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.299+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.299 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.300+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.299 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.301+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.299 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.301+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.300 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.302+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.300 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.302+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.300 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.302+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.300 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.303+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.300 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.304+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.305+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.306+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.306+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.307+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.307+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.307+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.302 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.309+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.303 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.309+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.303 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.310+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.303 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.311+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.303 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.311+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.303 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.311+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.304 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.311+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.304 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.312+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.305 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.312+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.306 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.313+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.307 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.314+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.307 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.314+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.308 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.315+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.308 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.316+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.308 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.316+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.309 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.316+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.309 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.316+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.309 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.317+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.310 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.317+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.311 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.318+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.311 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.318+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.311 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.318+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.311 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.319+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.311 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.319+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.312 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.319+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.312 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.320+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.312 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.320+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.313 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.321+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.313 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.322+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.315 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.323+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.316 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.324+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.316 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.324+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.317 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.324+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.317 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.325+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.318 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.326+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.318 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.326+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.318 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.326+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.318 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.326+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.318 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.327+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.319 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.328+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.323 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.330+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.324 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.331+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.325 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.332+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.326 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.332+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.326 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.333+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.326 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.334+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.329 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.335+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.329 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.335+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.330 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.337+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.330 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.337+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.330 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.337+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.331 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.338+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.331 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.338+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.331 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.338+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.331 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.339+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.333 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.339+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.333 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.340+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.334 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.341+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.334 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.341+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.334 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.341+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.335 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.342+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.335 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.342+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.335 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.343+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.336 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.343+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.336 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.344+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.336 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.344+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.341 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.345+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.345 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.348+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.345 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.348+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.345 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.348+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.346 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.349+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.346 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.349+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.347 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.350+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.347 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.350+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.347 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.351+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.347 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.351+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.347 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.351+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.348 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.352+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.348 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.352+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.348 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.352+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.348 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.354+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.349 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.355+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.349 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.356+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.356+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.357+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.357+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.357+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.357+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.357+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.351 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.357+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.351 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.359+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.359+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.359+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.353 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.360+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.354 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.361+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.356 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.365+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.357 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.365+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.360 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.367+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.360 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.367+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.361 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.367+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.361 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.367+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.363 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.367+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.364 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.368+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.364 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.369+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.365 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.370+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.366 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.371+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.366 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.373+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.366 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.373+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.367 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.373+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.367 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.374+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.368 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.374+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.368 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.375+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.369 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.376+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.369 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.376+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.369 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.376+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.369 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.376+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.370 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.377+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.371 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.378+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.372 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.379+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.373 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.379+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.373 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.380+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.373 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.380+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.375 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.382+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.375 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.382+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.375 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.383+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.376 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.383+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.376 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.384+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.378 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.384+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.385+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.386+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.386+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.386+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.387+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.387+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.381 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.387+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.381 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.388+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.382 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.389+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.382 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.389+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.382 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.390+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.383 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.391+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.392+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.393+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.393+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.393+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.394+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.394+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.394+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.394+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.395+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.396+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.389 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.396+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.389 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.396+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.389 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.397+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.390 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.397+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.390 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.399+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.391 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.400+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.391 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.400+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.393 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.401+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.393 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.402+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.394 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.403+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.394 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.403+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.395 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.404+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.395 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.404+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.396 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.405+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.397 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.406+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.404 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.409+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.405 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.409+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.405 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.409+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.405 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.410+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.406 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.410+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.407 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.411+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.407 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.412+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.407 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.412+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.407 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.412+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.407 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.413+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.409 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.414+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.409 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.416+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.409 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.416+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.409 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.416+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.409 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.416+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.410 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.417+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.410 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.417+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.410 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.418+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.411 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.419+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.411 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.419+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.411 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.419+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.411 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.420+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.416 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.421+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.416 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.422+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.416 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.422+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.417 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.424+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.417 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.425+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.418 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.425+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.419 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.428+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.420 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.428+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.420 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.428+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.420 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.428+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.421 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.429+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.421 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.429+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.421 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.430+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.422 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.431+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.424 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.432+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.424 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.433+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.424 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.433+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.425 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.433+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.425 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.434+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.426 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.434+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.426 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.435+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.426 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.435+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.427 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.436+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.428 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.436+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.429 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.438+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.429 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.438+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.430 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.438+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.432 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.440+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.432 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.440+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.432 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.440+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.433 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.441+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.433 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.441+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.433 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.442+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.433 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.442+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.434 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.443+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.434 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.443+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.443+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.444+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.444+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.436 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.444+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.437 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.445+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.437 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.446+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.437 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.446+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.438 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.447+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.443 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.447+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.444 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.448+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.444 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.448+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.445 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.449+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.445 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.449+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.445 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.450+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.445 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.450+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.446 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.451+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.446 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.451+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.447 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.452+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.448 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.452+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.448 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.452+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.448 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.453+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.449 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.454+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.449 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.455+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.449 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.456+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.450 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.456+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.450 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.456+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.450 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.456+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.451 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.456+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.451 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.456+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.451 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.457+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.451 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.458+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.451 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.458+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.452 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.459+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.452 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.460+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.452 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.460+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.453 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.461+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.453 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.461+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.455 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.461+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.455 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.462+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.456 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.463+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.456 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.463+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.456 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.463+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.457 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.464+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.458 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.464+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.458 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.465+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.458 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.465+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.459 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.466+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.459 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.466+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.460 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.467+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.460 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.468+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.460 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.469+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.461 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.469+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.461 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.469+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.461 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.470+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.462 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.470+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.462 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.471+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.463 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.472+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.463 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.472+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.464 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.472+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.464 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.473+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.465 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.474+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.465 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.474+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.466 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.474+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.466 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.475+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.467 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.476+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.470 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.476+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.471 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.477+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.471 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.478+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.472 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.479+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.472 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.479+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.472 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.481+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.473 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.481+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.474 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.481+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.474 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.482+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.474 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.482+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.474 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.482+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.475 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.483+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.475 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.483+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.475 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.484+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.477 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.484+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.478 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.484+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.478 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.485+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.478 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.486+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.480 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.488+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.480 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.488+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.481 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.490+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.481 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.490+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.482 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.490+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.482 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.490+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.482 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.491+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.483 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.492+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.484 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.492+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.484 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.493+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.485 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.493+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.485 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.494+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.488 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.496+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.488 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.496+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.488 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.496+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.489 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.496+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.489 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.496+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.493 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.498+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.494 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.499+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.494 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.499+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.494 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.499+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.495 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.499+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.495 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.501+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.495 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.503+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.496 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.503+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.496 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.504+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.496 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.504+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.497 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.504+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.498 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.505+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.499 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.506+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.499 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.506+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.499 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.506+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.500 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.507+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.500 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.508+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.501 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.508+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.501 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.508+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.501 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.509+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.501 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.509+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.504 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.509+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.504 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.510+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.505 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.510+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.505 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.511+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.505 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.512+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.505 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.512+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.506 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.513+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.506 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.514+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.506 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.514+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.507 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.514+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.507 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.514+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.507 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.514+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.508 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.515+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.508 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.515+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.508 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.517+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.509 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.517+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.509 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.518+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.510 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.518+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.511 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.519+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.511 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.520+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.511 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.520+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.512 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.521+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.512 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.521+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.512 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.521+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.513 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.522+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.513 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.522+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.514 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.522+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.514 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.523+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.515 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.524+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.515 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.524+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.516 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.525+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.517 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.525+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.517 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.526+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.519 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.527+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.519 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.528+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.520 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.529+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.520 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.529+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.522 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.531+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.522 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.531+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.523 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.531+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.523 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.532+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.523 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.532+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.524 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.532+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.528 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.534+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.528 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.534+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.528 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.535+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.530 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.536+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.530 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.536+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.530 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.536+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.531 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.537+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.532 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.538+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.532 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.538+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.532 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.538+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.532 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.538+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.532 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.539+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.532 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.540+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.532 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.540+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.535 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.543+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.535 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.543+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.535 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.544+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.536 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.544+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.536 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.544+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.536 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.545+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.537 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.545+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.541 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.546+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.542 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.546+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.542 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.547+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.543 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.547+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.543 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.548+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.543 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.549+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.543 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.549+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.544 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.550+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.544 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.551+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.544 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.551+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.544 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.552+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.544 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.552+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.546 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.553+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.548 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.553+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.549 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.553+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.549 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.553+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.549 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.554+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.550 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.555+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.552 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.556+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.552 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.557+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.552 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.557+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.553 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.557+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.558+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.558+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.558+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.558+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.559+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.560+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.555 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.560+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.555 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.561+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.561+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.562+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.562+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.562+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.563+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.563+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.557 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.564+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.557 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.565+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.560 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.566+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.560 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.568+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.561 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.569+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.567 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.570+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.567 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.570+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.568 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.573+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.569 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.573+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.569 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.573+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.569 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.574+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.569 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.574+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.569 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.574+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.570 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.575+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.570 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.575+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.570 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.576+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.570 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.577+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.570 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.577+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.571 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.579+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.580+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.581+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.573 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.581+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.573 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.581+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.573 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.581+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.578 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.583+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.578 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.583+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.578 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.583+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.584+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.584+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.585+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.585+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.580 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.585+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.580 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.586+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.586+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.587+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.588+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.588+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.589+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.582 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.590+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.582 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.590+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.582 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.591+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.582 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.591+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.585 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.592+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.587 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.592+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.588 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.593+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.590 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.594+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.591 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.596+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.591 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.596+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.591 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.598+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.592 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.598+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.592 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.598+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.592 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.599+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.593 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.601+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.593 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.601+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.593 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.601+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.593 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.601+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.602+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.602+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.602+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.603+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.595 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.604+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.596 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.604+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.596 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.605+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.599 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.607+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.599 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.607+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.600 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.608+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.601 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.608+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.601 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.609+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.601 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.610+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.602 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.610+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.602 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.610+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.602 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.611+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.603 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.611+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.604 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.613+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.604 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.613+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.605 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.613+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.605 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.613+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.605 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.614+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.606 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.614+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.609 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.616+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.609 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.616+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.610 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.617+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.610 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.618+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.610 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.618+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.611 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.618+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.611 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.619+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.612 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.619+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.612 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.620+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.613 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.620+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.613 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.621+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.614 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.621+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.614 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.622+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.614 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.622+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.615 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.624+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.616 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.624+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.616 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.625+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.617 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.626+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.618 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.627+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.619 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.628+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.619 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.628+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.619 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.628+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.620 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.628+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.620 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.628+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.620 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.629+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.625 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.631+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.625 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.631+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.628 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.632+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.628 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.632+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.629 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.633+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.629 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.633+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.629 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.633+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.629 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.634+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.630 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.634+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.630 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.635+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.635+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.636+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.637+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.638+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.639+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.639+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.634 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.640+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.634 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.641+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.634 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.641+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.635 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.642+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.635 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.642+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.635 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.643+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.636 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.644+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.636 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.644+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.638 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.646+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.647+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.648+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.648+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.642 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.649+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.642 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.649+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.643 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.650+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.643 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.651+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.643 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.651+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.643 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.652+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.644 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.652+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.644 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.652+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.644 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.653+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.648 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.654+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.648 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.656+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.648 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.656+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.649 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.656+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.649 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.656+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.649 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.657+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.650 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.657+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.650 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.658+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.650 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.659+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.652 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.659+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.653 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.659+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.653 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.659+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.654 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.661+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.654 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.661+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.654 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.661+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.655 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.662+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.655 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.662+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.655 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.662+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.655 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.663+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.657 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.663+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.657 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.664+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.658 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.664+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.658 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.664+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.660 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.665+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.661 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.666+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.661 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.667+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.661 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.667+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.664 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.668+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.664 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.668+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.664 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.669+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.669+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.669+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.670+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.670+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.666 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.670+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.666 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.670+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.666 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.671+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.666 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.671+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.666 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.671+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.667 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.673+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.667 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.673+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.667 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.673+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.667 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.674+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.668 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.675+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.668 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.676+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.668 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.677+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.669 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.677+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.670 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.679+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.671 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.680+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.671 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.680+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.672 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.681+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.673 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.682+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.674 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.683+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.674 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.683+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.675 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.683+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.675 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.684+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.676 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.684+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.676 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.685+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.677 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.685+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.677 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.685+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.677 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.686+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.677 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.686+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.678 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.686+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.678 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.687+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.679 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.688+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.680 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.689+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.681 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.690+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.683 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.691+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.683 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.691+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.684 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.691+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.684 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.692+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.684 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.692+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.684 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.693+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.686 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.694+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.686 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.695+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.688 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.695+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.689 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.697+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.691 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.700+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.691 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.700+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.700+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.700+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.701+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.697 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.702+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.697 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.702+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.697 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.702+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.698 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.703+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.703+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.704+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.704+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.704+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.704+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.704+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.705+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.706+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.706+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.703 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.707+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.704 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.708+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.704 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.709+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.704 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.710+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.704 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.710+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.704 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.710+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.705 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.713+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.706 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.713+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.706 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.713+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.706 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.714+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.706 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.714+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.707 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.715+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.707 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.716+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.708 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.717+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.708 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.717+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.709 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.717+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.709 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.717+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.709 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.717+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.710 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.718+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.710 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.719+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.711 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.720+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.711 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.720+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.712 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.721+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.712 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.721+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.713 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.721+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.713 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.722+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.714 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.723+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.715 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.724+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.717 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.724+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.717 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.725+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.717 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.725+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.718 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.725+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.721 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.726+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.721 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.727+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.721 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.728+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.721 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.728+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.722 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.728+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.722 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.728+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.722 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.729+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.723 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.730+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.723 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.731+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.723 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.732+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.723 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.732+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.724 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.733+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.724 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.733+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.725 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.733+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.726 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.734+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.728 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.735+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.728 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.736+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.729 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.737+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.729 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.737+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.730 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.738+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.730 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.739+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.732 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.741+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.732 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.741+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.733 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.742+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.734 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.742+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.734 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.743+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.736 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.745+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.737 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.745+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.737 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.745+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.738 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.746+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.738 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.746+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.738 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.746+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.741 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.748+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.741 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.748+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.741 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.749+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.742 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.750+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.742 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.751+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.743 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.751+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.743 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.751+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.743 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.751+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.743 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.752+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.744 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.753+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.744 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.753+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.745 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.754+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.745 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.754+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.746 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.754+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.746 INFO:teuthology.orchestra.run.smithi120.stdout: 16 16 16146 16130 3.9377 3.30469 0.0160657 0.0158599 2020-11-16T21:31:24.747 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.755+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.747 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.755+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.749 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.757+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.749 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.757+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.749 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.757+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.750 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.758+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.750 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.758+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.750 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.759+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.750 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.759+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.751 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.760+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.751 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.760+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.752 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.760+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.752 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.760+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.755 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.761+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.755 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.762+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.757 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.762+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.757 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.763+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.758 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.763+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.758 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.764+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.758 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.764+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.758 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.764+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.759 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.765+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.760 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.766+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.760 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.766+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.760 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.766+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.760 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.766+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.767+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.767+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.767+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.767+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.767+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.768+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.769+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.762 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.769+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.762 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.770+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.762 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.770+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.763 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.772+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.764 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.773+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.764 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.773+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.771 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.776+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.771 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.776+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.774 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.777+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.774 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.777+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.774 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.777+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.775 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.778+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.775 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.779+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.775 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.779+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.775 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.779+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.779+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.780+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.778 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.780+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.778 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.781+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.778 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.781+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.778 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.782+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.780 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.783+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.780 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.783+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.780 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.784+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.780 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.784+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.780 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.784+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.780 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.784+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.780 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.785+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.781 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.785+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.781 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.786+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.781 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.787+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.782 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.787+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.782 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.787+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.784 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.788+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.784 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.789+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.784 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.789+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.785 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.791+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.785 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.791+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.785 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.793+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.786 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.793+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.786 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.794+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.786 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.794+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.788 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.795+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.788 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.795+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.789 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.795+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.790 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.795+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.790 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.796+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.796+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.797+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.797+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.798+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.799+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.800+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.800+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.793 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.801+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.793 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.801+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.793 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.802+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.794 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:24.802+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:24.794 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.802+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.797 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.803+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.797 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.803+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.798 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.805+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.799 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.806+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.800 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.806+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.800 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.806+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.800 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.807+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.800 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.808+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.800 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.808+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.800 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.808+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.801 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.808+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.801 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.808+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.801 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.809+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.802 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.810+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.804 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.810+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.804 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.811+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.804 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.813+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.805 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.813+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.805 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.814+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.805 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.814+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.807 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.816+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.807 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.816+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.812 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.817+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.812 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.818+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.813 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.818+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.814 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.819+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.814 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.819+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.814 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.819+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.814 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.819+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.814 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.820+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.815 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.820+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.815 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.820+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.815 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.820+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.815 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.821+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.815 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.821+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.816 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.823+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.818 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.824+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.818 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.824+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.819 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.826+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.819 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.826+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.819 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.826+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.820 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.827+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.821 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.829+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.821 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.830+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.821 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.830+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.822 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.830+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.822 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.830+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.822 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.831+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.826 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.832+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.826 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.832+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.826 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.833+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.827 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.833+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.828 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.834+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.828 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.834+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.828 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.835+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.828 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.835+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.828 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.836+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.828 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.836+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.829 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.837+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.829 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.838+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.829 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.838+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.838+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.839+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.839+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.831 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.840+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.834 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.841+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.834 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.841+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.834 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.841+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.836 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.842+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.836 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.842+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.836 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.842+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.836 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.843+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.837 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.843+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.843+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.844+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.844+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.845+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.845+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.845+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.846+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.839 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.846+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.839 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.846+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.844 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.848+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.845 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.848+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.845 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.849+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.846 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.849+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.846 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.853+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.848 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.855+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.848 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.855+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.848 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.855+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.848 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.856+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.848 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.856+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.851 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.857+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.851 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.857+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.858+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.859+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.859+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.860+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.860+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.853 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.861+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.853 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.861+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.853 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.862+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.854 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.862+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.854 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.863+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.855 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.863+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.855 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.864+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.855 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.864+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.856 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.865+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.856 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.865+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.857 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.866+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.859 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.866+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.859 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.867+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.859 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.867+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.859 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.868+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.860 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.868+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.860 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.868+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.860 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.869+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.861 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.869+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.861 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.870+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.862 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.870+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.862 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.871+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.867 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.871+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.867 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.872+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.873+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.873+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.873+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.869 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.874+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.869 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.875+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.869 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.876+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.870 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.877+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.870 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.877+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.870 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.877+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.872 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.877+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.873 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.878+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.873 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.878+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.874 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.878+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.874 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.879+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.874 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.879+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.874 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.880+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.875 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.880+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.876 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.881+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.876 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.881+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.876 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.881+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.876 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.883+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.876 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.883+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.876 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.883+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.877 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.884+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.877 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.884+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.877 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.884+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.878 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.885+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.878 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.885+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.878 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.886+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.878 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.886+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.880 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.887+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.880 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.888+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.881 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.888+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.881 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.888+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.882 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.890+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.882 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.890+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.883 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.891+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.883 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.891+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.883 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.891+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.884 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.892+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.885 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.893+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.885 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.894+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.886 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.894+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.886 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.894+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.886 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.894+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.886 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.894+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.887 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.895+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.889 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.896+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.889 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.897+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.890 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.899+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.893 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.900+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.894 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.900+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.894 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.901+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.894 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.902+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.894 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.902+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.895 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.903+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.895 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.904+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.896 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.904+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.896 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.904+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.896 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.905+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.897 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.906+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.898 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.906+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.898 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.906+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.898 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.907+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.899 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.907+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.899 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.907+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.899 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.907+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.899 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.908+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.900 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.909+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.902 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.910+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.904 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.911+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.904 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.911+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.904 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.912+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.905 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.913+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.905 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.913+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.905 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.913+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.905 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.914+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.907 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.915+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.908 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.916+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.908 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.916+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.909 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.917+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.909 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.917+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.910 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.919+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.911 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.920+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.911 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.920+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.912 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.921+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.912 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.921+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.913 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.921+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.922+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.923+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.924+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.924+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.924+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.924+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.924+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.925+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.918 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.926+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.918 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.927+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.918 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.927+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.919 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.928+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.923 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.928+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.924 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.930+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.924 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.930+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.924 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.931+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.925 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.931+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.925 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.932+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.925 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.932+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.926 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.933+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.926 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.934+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.926 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.935+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.927 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.935+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.927 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.936+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.927 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.936+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.928 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.936+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.928 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.937+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.929 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.938+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.929 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.938+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.929 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.938+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.930 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.939+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.930 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.939+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.932 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.939+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.933 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.939+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.934 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.941+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.934 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.941+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.937 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.944+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.944+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.944+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.945+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.946+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.946+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.939 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.947+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.939 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.947+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.939 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.948+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.940 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.949+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.940 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.949+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.941 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.949+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.941 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.949+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.941 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.950+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.944 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.953+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.944 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.953+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.945 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.953+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.945 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.953+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.945 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.954+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.946 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.955+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.947 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.955+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.947 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.955+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.947 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.956+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.948 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.956+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.948 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.956+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.949 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.958+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.952 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.958+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.953 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.958+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.953 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.959+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.954 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.959+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.954 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.960+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.955 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.960+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.955 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.961+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.955 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.961+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.956 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.961+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.956 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.962+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.957 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.962+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.957 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.963+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.958 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.964+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.958 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.964+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.958 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.964+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.958 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.965+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.958 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.965+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.959 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.966+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.959 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.966+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.959 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.967+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.959 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.968+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.960 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.968+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.962 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.968+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.962 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.969+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.962 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.970+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.962 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.970+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.963 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.971+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.963 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.971+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.967 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.972+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.967 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.972+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.967 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.973+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.967 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.974+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.968 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.974+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.969 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.974+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.969 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.975+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.975+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.976+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.977+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.978+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.971 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.978+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.971 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.978+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.971 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.979+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.972 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.979+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.972 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.980+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.975 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.980+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.975 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.980+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.976 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.983+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.976 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.983+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.976 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.984+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.977 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.984+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.977 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.985+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.978 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.985+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.979 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.985+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.979 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.986+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.979 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.988+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.980 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.988+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.980 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.989+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.984 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.990+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.985 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.990+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.985 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.991+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.985 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.991+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.985 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.991+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.985 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.992+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.986 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.994+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.986 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.994+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.987 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.994+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.987 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.995+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.987 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.995+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.989 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.996+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.990 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.996+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.990 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.996+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.990 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.996+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.991 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.997+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.991 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.997+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.991 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.998+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.992 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.998+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.992 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.999+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.992 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.999+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.992 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:24.999+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.993 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.000+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.993 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.001+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.993 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.001+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.993 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.001+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.993 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.001+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.995 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.002+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.995 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.002+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.996 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.003+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.996 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.004+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.996 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.004+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.996 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.005+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:24.997 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.005+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.003 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.009+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.004 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.010+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.005 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.010+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.005 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.011+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.005 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.011+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.005 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.011+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.005 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.011+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.005 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.011+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.006 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.012+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.012+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.013+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.014+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.014+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.014+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.015+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.008 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.015+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.010 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.016+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.010 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.017+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.011 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.017+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.011 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.017+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.015 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.018+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.015 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.019+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.019+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.020+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.020+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.021+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.017 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.021+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.017 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.022+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.022+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.023+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.023+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.023+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.023+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.023+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.019 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.024+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.019 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.025+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.019 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.026+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.020 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.026+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.022 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.027+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.022 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.028+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.029+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.031+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.024 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.031+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.024 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.031+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.024 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.031+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.025 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.032+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.025 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.033+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.025 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.033+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.025 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.033+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.025 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.033+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.026 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.034+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.026 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.034+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.031 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.035+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.031 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.036+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.031 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.036+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.032 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.036+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.032 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.037+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.037+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.039+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.039+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.040+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.041+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.034 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.041+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.034 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.041+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.034 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.042+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.043+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.043+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.043+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.043+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.044+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.045+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.045+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.047+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.047+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.047+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.047+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.042 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.049+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.042 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.049+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.042 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.050+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.050+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.050+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.052+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.052+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.044 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.053+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.045 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.053+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.047 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.053+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.047 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.054+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.047 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.054+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.047 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.054+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.047 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.055+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.047 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.055+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.050 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.059+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.051 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.059+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.051 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.060+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.052 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.060+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.052 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.060+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.052 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.061+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.053 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.062+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.053 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.062+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.053 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.062+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.056 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.065+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.056 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.065+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.057 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.065+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.057 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.066+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.058 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.066+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.058 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.067+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.058 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.067+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.059 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.068+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.068+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.060 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.068+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.061 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.069+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.061 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.069+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.062 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.070+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.062 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.071+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.063 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.071+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.066 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.074+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.066 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.074+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.066 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.074+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.067 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.075+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.068 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.075+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.068 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.076+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.068 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.076+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.069 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.076+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.070 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.078+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.070 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.078+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.071 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.079+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.072 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.080+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.072 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.080+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.072 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.080+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.072 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.080+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.072 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.081+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.073 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.081+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.074 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.082+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.076 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.084+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.076 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.084+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.076 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.084+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.078 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.085+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.079 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.085+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.079 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.087+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.079 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.087+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.080 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.088+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.081 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.090+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.082 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.090+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.082 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.090+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.082 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.091+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.083 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.091+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.086 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.093+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.086 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.093+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.086 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.093+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.087 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.093+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.087 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.093+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.087 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.094+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.087 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.095+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.088 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.096+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.088 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.096+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.089 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.097+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.091 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.097+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.091 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.099+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.092 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.100+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.092 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.100+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.092 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.100+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.092 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.101+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.096 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.102+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.096 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.102+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.096 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.104+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.097 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.104+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.097 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.104+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.097 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.105+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.101 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.107+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.101 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.107+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.101 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.107+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.102 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.108+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.102 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.109+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.102 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.109+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.103 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.109+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.103 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.111+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.103 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.111+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.103 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.111+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.104 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.112+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.104 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.112+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.104 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.113+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.105 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.113+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.107 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.114+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.107 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.114+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.108 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.115+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.108 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.115+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.111 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.116+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.116 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.118+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.116 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.119+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.116 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.119+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.117 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.119+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.117 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.119+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.117 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.119+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.118 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.119+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.118 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.119+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.118 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.121+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.118 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.121+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.119 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.122+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.119 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.122+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.119 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.126+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.120 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.126+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.120 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.128+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.121 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.128+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.121 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.128+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.121 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.129+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.121 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.129+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.122 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.130+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.123 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.130+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.124 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.130+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.124 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.130+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.125 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.132+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.125 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.132+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.125 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.132+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.126 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.133+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.126 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.134+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.126 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.135+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.127 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.136+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.127 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.136+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.128 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.136+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.132 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.137+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.137 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.138+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.137 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.139+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.138 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.140+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.138 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.141+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.138 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.141+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.139 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.141+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.142+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.143+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.143+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.144+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.144+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.140 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.144+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.141 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.146+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.141 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.147+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.147+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.147+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.148+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.148+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.149+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.149+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.142 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.150+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.146 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.152+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.146 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.152+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.147 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.153+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.147 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.154+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.147 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.154+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.147 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.155+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.148 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.156+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.148 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.156+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.148 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.156+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.152 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.158+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.152 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.159+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.153 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.160+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.153 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.160+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.153 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.161+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.153 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.161+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.154 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.162+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.154 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.162+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.154 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.162+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.155 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.162+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.155 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.163+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.155 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.164+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.156 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.164+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.156 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.165+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.157 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.166+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.157 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.166+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.158 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.167+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.161 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.167+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.161 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.168+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.172+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.172+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.172+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.172+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.167 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.172+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.168 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.173+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.170 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.173+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.171 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.174+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.171 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.175+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.171 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.175+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.172 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.175+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.172 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.176+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.172 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.176+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.177+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.178+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.178+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.179+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.179+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.173 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.180+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.174 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.180+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.174 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.181+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.175 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.181+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.175 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.182+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.175 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.183+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.175 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.183+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.175 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.183+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.179 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.184+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.180 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.185+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.180 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.186+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.180 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.186+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.181 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.187+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.182 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.188+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.182 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.189+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.182 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.189+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.182 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.189+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.182 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.189+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.183 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.189+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.185 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.189+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.186 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.190+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.186 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.191+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.186 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.192+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.187 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.192+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.187 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.192+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.187 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.192+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.187 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.194+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.188 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.194+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.188 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.195+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.188 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.196+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.189 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.198+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.190 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.199+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.190 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.199+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.191 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.199+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.191 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.199+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.191 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.200+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.192 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.200+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.192 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.201+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.193 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.202+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.195 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.203+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.196 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.204+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.197 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.206+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.198 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.207+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.199 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.208+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.200 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.209+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.200 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.209+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.201 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.209+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.201 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.209+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.201 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.210+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.203 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.212+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.204 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.212+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.204 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.213+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.205 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.213+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.205 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.214+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.205 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.214+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.206 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.215+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.207 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:25.216+0000 7f6e23204700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6800/21985 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:25.208 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.215+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.208 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.216+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.208 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.216+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.213 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.218+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.213 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.218+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.213 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.219+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.214 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.219+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.214 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.219+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.214 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.221+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.215 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.222+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.215 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.222+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.216 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.223+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.216 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.224+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.216 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.224+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.216 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.224+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.217 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.225+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.218 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.227+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.218 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.227+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.219 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.228+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.220 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.228+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.220 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.229+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.222 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.229+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.224 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.230+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.224 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.231+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.224 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.231+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.224 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.231+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.224 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.231+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.224 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.231+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.225 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.231+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.225 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.232+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.226 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.226 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.226 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.226 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f54b8b11700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) discard objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f54b8b11700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) discard objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f54b8b11700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) discard objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f54b8b11700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) discard objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.233+0000 7f54b8b11700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) discard objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.227 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.234+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.229 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.235+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.229 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.235+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.229 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.235+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.229 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.236+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.230 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.236+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.230 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.236+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.230 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.237+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.239+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.239+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.231 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.240+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.232 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.240+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.232 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.241+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.232 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.241+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.233 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.241+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.233 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.242+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.234 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.243+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.235 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.243+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.235 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.243+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.235 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.244+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.236 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.244+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.236 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.245+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.236 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.245+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.237 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.246+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.237 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.246+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.238 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.246+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.238 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.246+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.239 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.247+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.239 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.247+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.239 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.248+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.240 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.248+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.240 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.249+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.240 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.249+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.244 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.250+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.245 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.250+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.245 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.252+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.246 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.253+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.246 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.253+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.246 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.254+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.247 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.255+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.247 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.256+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.248 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.256+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.248 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.257+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.249 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.258+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.250 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.258+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.250 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.258+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.250 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.259+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.251 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.259+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.251 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.259+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.251 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.259+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.251 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.260+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.252 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.261+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.253 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.261+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.253 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.262+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.255 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.263+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.255 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.263+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.260 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.267+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.260 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.267+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.260 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.267+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.261 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.268+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.261 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.268+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.262 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.269+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.262 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.269+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.262 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.269+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.262 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.270+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.263 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.271+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.263 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.271+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.264 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.271+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.264 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.272+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.265 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.272+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.265 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.272+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.265 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.273+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.265 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.274+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.266 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.274+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.272 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.276+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.277+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.278+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.278+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.278+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.275 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.279+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.279+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.280+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.280+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.281+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.281+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.281+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.276 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.281+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.279 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.284+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.280 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.285+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.280 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.285+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.281 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.286+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.281 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.287+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.281 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.287+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.281 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.287+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.282 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.287+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.282 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.288+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.283 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.288+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.283 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.288+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.283 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.289+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.284 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.290+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.284 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.291+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.284 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.291+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.284 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.292+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.284 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.292+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.293+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.285 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.293+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.287 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.296+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.288 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.296+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.288 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.296+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.288 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.297+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.289 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.297+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.292 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.299+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.292 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.299+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.293 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.300+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.293 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.300+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.293 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.300+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.294 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.301+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.294 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.302+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.294 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.302+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.297 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.304+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.297 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.304+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.297 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.305+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.297 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.305+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.298 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.305+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.298 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.306+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.298 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.306+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.298 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.306+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.307+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.307+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.301 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.307+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.303 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.308+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.304 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.309+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.304 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.310+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.305 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.311+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.305 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.312+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.306 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.312+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.306 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.312+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.306 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.312+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.307 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.314+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.307 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.315+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.307 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.315+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.307 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.315+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.308 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.316+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.308 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.317+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.309 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.317+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.311 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.318+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.311 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.318+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.312 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.318+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.312 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.321+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.313 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.322+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.313 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.322+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.313 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.322+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.318 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.322+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.319 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.325+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.319 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.325+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.319 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.325+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.320 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.325+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.320 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.326+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.321 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.329+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.321 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.330+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.322 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.330+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.322 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.330+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.322 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.330+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.323 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.331+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.323 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.331+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.323 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.331+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.323 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.331+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.326 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.332+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.327 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.332+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.327 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.332+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.328 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.333+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.328 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.334+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.328 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.334+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.329 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.335+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.329 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.335+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.329 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.335+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.329 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.337+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.330 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.338+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.331 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.340+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.335 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.342+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.336 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.342+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.336 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.343+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.337 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.343+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.337 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.343+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.337 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.344+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.338 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.346+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.339 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.347+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.339 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.347+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.339 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.347+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.340 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.348+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.340 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.348+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.340 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.349+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.342 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.351+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.343 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.351+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.343 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.351+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.343 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.352+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.343 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.352+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.344 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.353+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.344 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.353+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.349 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.354+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.349 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.354+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.349 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.354+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.355+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.356+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.350 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.356+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.351 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.357+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.357+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.357+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.358+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.359+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.359+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.352 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.359+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.353 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.359+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.353 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.360+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.353 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.360+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.353 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.360+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.354 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.361+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.354 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.362+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.354 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.362+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.355 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.364+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.355 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.364+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.356 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.365+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.357 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.365+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.358 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.366+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.359 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.367+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.360 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.367+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.360 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.368+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.360 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.368+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.361 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.369+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.361 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.370+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.363 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.371+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.363 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.371+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.363 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.372+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.364 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.372+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.365 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.373+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.365 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.373+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.366 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.373+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.366 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.373+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.366 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.374+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.366 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.375+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.371 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.375+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.371 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.375+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.371 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.376+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.372 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.376+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.372 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.377+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.372 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.377+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.373 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.379+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.373 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.379+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.376 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.380+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.376 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.381+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.377 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.381+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.377 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.382+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.378 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.382+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.378 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.385+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.379 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.385+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.379 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.385+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.387+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.387+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.380 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.387+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.381 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.388+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.381 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.388+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.381 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.389+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.381 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.389+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.382 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.389+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.382 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.390+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.382 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.391+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.385 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.391+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.385 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.392+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.392+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.392+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.392+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.386 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.393+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.387 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.394+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.387 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.394+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.387 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.395+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.396+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.396+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.388 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.397+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.391 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.398+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.391 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.399+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.391 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.399+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.392 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.399+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.392 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.399+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.392 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.400+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.392 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.401+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.392 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.401+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.393 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.402+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.393 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.402+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.393 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.402+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.395 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.404+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.395 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.404+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.396 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.405+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.397 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.405+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.399 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.408+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.400 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.408+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.400 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.409+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.400 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.409+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.400 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.409+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.401 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.409+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.401 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.409+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.402 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.411+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.407 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.413+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.408 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.413+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.408 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.414+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.408 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.414+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.408 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.414+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.409 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.415+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.411 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.418+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.412 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.419+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.412 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.419+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.412 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.419+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.412 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.420+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.412 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.420+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.413 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.420+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.413 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.421+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.413 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.421+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.414 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.421+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.414 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.422+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.414 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.422+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.420 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.423+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.420 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.423+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.421 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.424+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.421 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.424+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.421 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.424+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.422 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.426+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.422 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.426+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.422 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.426+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.423 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.426+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.423 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.427+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.423 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.428+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.423 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.428+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.424 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.428+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.425 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.429+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.425 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.429+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.425 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.429+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.425 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.429+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.426 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.430+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.426 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.430+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.427 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.433+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.427 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.434+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.427 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.434+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.428 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.435+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.428 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.436+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.428 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.436+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.429 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.436+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.429 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.437+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.429 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.438+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.432 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.438+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.433 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.439+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.433 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.439+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.434 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.439+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.434 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.439+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.440+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.440+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.441+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.441+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.435 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.442+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.436 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.442+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.436 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.442+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.436 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.443+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.436 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.443+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.436 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.444+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.437 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.444+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.437 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.445+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.437 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.446+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.438 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.447+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.439 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.448+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.440 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.448+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.440 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.448+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.440 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.449+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.442 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.451+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.443 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.452+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.443 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.452+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.443 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.452+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.444 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.453+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.445 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.454+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.445 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.454+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.446 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.454+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.446 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.455+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.447 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.455+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.447 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.455+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.447 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.456+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.447 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.456+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.448 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.457+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.449 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.457+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.449 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.458+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.450 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.458+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.450 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.459+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.450 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.459+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.456 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.460+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.456 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.462+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.458 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.462+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.458 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.462+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.459 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.463+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.459 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.467+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.460 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.467+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.460 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.468+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.461 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.469+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.461 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.470+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.462 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.470+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.462 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.470+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.462 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.470+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.462 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.470+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.462 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.471+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.463 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.472+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.463 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.472+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.464 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.473+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.465 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.473+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.465 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.474+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.466 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.475+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.467 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.475+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.468 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.477+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.468 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.477+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.469 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.478+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.476 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.479+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.477 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.479+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.477 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.480+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.477 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.480+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.478 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.481+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.478 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.481+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.479 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.482+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.479 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.482+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.479 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.483+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.479 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.483+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.480 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.483+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.480 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.484+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.480 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.484+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.480 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.485+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.481 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.485+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.481 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.487+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.481 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.487+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.482 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.488+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.482 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.488+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.482 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.488+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.485 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.491+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.486 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.491+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.486 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.491+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.486 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.491+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.486 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.492+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.487 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.493+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.487 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.493+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.487 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.493+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.487 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.494+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.487 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.494+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.487 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.495+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.488 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.496+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.491 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.499+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.492 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.500+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.493 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.500+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.493 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.500+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.493 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.501+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.493 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.501+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.494 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.502+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.494 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.502+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.494 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.503+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.495 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.503+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.495 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.504+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.496 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.505+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.497 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.506+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.499 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.508+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.499 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.508+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.500 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.508+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.500 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.509+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.500 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.509+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.501 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.510+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.502 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.510+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.505 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.511+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.506 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.512+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.506 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.512+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.506 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.513+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.511 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.513+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.511 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.514+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.512 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.514+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.512 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.516+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.512 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.516+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.513 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.516+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.513 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.517+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.514 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.519+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.514 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.519+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.514 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.519+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.515 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.520+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.515 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.520+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.515 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.521+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.515 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.521+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.515 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.522+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.516 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.522+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.516 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.523+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.516 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.523+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.517 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.523+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.517 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.523+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.519 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.524+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.520 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.526+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.520 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.526+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.520 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.526+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.521 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.526+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.521 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.527+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.521 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.527+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.521 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.530+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.522 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.530+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.522 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.531+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.523 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.532+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.523 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.532+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.526 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.532+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.527 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.533+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.527 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.533+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.527 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.534+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.527 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.534+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.528 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.535+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.530 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.535+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.530 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.535+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.533 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.536+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.534 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.537+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.535 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.539+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.535 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.539+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.536 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.539+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.536 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.542+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.537 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.542+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.537 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.542+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.537 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.542+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.538 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.546+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.538 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.547+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.539 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.547+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.539 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.548+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.539 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.548+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.540 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.548+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.542 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.549+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.543 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.549+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.543 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.550+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.543 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.551+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.546 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.551+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.546 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.551+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.547 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.552+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.547 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.554+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.547 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.554+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.547 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.554+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.548 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.555+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.548 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.555+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.549 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.556+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.549 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.557+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.549 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.557+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.549 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.557+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.550 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.558+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.550 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.558+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.551 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.559+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.551 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.559+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.551 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.560+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.551 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.560+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.552 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.561+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.553 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.562+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.563+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.554 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.563+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.555 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.563+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.555 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.563+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.555 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.563+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.564+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.564+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.565+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.560 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.567+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.561 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.569+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.561 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.569+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.561 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.570+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.562 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.570+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.562 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.570+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.562 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.571+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.563 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.571+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.563 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.572+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.563 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.572+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.564 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.572+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.564 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.573+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.565 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.573+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.565 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.574+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.571 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.576+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.571 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.576+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.571 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.576+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.577+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.577+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.578+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.578+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.580+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.580+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.573 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.580+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.573 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.580+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.573 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.582+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.574 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.582+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.574 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.582+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.574 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.583+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.578 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.584+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.578 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.584+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.585+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.585+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.586+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.587+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.587+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.587+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.579 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.587+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.580 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.588+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.580 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.588+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.588+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.589+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.589+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.581 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.590+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.582 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.591+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.583 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.591+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.585 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.594+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.585 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.594+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.586 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.594+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.586 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.594+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.586 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.594+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.591 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.595+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.592 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.595+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.592 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.595+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.592 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.596+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.593 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.597+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.593 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.597+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.598+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.599+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.599+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.594 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.600+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.595 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.601+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.595 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.601+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.595 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.601+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.596 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.603+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.596 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.604+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.598 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.604+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.598 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.604+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.599 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.604+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.599 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.605+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.600 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.605+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.600 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.606+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.600 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.606+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.600 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.607+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.600 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.607+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.601 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.608+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.601 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.609+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.601 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.609+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.602 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.610+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.607 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.612+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.608 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.613+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.611 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.614+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.611 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.615+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.611 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.615+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.611 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.615+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.612 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.615+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.612 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.616+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.612 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.616+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.614 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.617+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.614 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.617+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.615 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.620+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.616 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.620+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.616 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.620+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.618 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.621+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.618 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.622+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.619 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.623+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.619 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.624+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.619 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.624+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.620 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.624+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.620 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.624+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.620 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.626+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.621 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.626+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.621 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.627+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.621 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.628+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.621 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.628+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.621 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.628+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.621 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.629+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.622 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.629+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.622 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.630+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.622 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.630+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.623 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.631+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.623 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.631+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.623 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.632+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.624 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.633+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.625 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.633+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.626 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.634+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.626 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.634+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.627 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.635+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.627 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.636+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.628 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.636+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.628 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.637+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.630 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.637+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.638+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.638+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.631 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.638+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.632 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.639+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.632 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.639+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.633 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.639+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.633 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.641+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.634 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.642+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.634 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.642+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.634 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.643+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.638 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.644+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.638 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.644+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.638 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.644+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.639 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.645+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.639 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.646+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.640 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.646+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.640 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.646+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.640 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.646+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.640 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.647+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.640 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.647+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.640 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.648+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.649+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.649+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.650+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.644 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.651+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.644 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.651+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.644 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.651+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.644 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.652+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.645 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.652+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.645 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.653+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.646 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.654+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.646 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.654+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.646 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.654+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.647 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.655+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.647 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.655+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.650 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.656+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.651 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.657+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.651 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.658+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.652 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.658+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.652 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.659+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.652 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.659+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.653 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.660+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.653 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.660+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.653 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.661+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.653 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.661+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.653 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.661+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.654 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.662+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.654 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.663+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.655 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.663+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.655 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.664+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.655 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.664+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.656 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.664+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.656 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.665+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.657 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.665+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.657 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.665+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.657 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.665+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.658 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.666+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.658 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.667+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.658 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.667+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.659 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.668+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.660 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.669+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.660 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.669+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.662 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.670+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.664 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.671+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.672+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.672+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.672+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.673+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.666 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.673+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.666 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.674+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.667 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.675+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.667 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.675+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.669 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.676+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.669 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.676+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.669 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.676+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.670 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.678+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.670 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.678+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.670 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.678+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.671 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.678+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.671 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.678+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.671 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.680+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.673 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.681+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.673 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.682+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.674 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.682+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.674 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.682+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.674 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.682+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.676 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.683+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.677 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.684+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.677 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.685+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.677 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.685+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.678 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.685+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.678 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.685+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.678 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.686+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.678 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.687+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.679 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.688+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.679 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.688+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.680 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.688+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.681 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.689+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.681 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.690+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.682 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.691+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.683 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.691+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.683 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.691+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.683 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.692+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.684 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.692+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.688 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.692+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.688 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.693+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.689 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.694+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.689 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.694+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.690 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.694+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.690 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.695+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.690 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.695+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.690 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.696+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.690 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.696+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.691 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.696+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.691 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.697+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.691 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.698+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.698+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.699+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.700+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.700+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.692 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.700+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.693 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.701+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.693 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.702+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.694 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.703+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.695 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.703+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.695 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.704+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.695 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.704+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.698 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.704+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.698 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.705+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.706+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.706+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.706+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.706+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.707+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.707+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.708+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.708+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.700 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.708+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.701 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.708+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.701 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.709+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.701 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.709+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.702 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.711+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.706 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.714+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.706 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.714+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.709 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.714+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.709 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.714+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.709 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.716+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.710 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.716+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.710 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.717+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.711 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.718+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.711 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.718+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.711 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.719+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.711 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.719+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.712 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.720+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.713 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.720+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.713 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.720+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.714 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.720+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.714 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.721+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.714 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.722+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.715 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.722+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.715 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.723+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.715 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.723+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.715 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.724+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.716 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.724+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.716 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.724+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.716 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.725+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.717 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.725+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.717 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.725+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.718 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.726+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.718 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.726+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.718 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.726+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.719 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.727+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.719 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.728+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.719 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.728+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.723 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.730+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.723 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.731+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.724 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.731+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.724 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.731+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.724 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.732+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.724 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.732+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.725 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.733+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.725 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.733+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.727 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.734+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.728 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.735+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.728 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.735+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.728 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.736+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.729 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.736+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.729 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.737+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.729 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.737+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.729 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.738+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.730 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.738+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.733 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.741+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.733 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.741+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.733 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.741+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.733 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.742+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.734 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.742+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.737 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.743+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.737 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.743+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.738 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.743+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.738 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.744+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.739 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.745+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.739 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.746+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.739 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.747+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.741 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.748+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.742 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.748+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.742 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.748+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.742 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.749+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.743 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.751+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.743 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.751+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.744 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.752+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.744 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.752+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.745 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.752+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.745 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.754+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.745 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.754+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.746 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.754+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.746 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.754+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.746 INFO:teuthology.orchestra.run.smithi120.stdout: 17 16 16997 16981 3.90159 3.32422 0.0240574 0.0160091 2020-11-16T21:31:25.750 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.755+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.753 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.756+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.753 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.756+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.754 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.758+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.754 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.758+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.754 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.759+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.754 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.759+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.754 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.762+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.754 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.762+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.756 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.762+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.756 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.763+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.756 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.764+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.757 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.764+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.757 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.764+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.757 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.765+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.757 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.765+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.757 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.765+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.766+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.767+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.767+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.768+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.761 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.769+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.762 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.769+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.762 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.770+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.763 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.772+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.763 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.772+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.763 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.772+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.764 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.772+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.764 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.773+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.765 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.773+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.765 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.773+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.765 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.773+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.766 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.775+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.766 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.775+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.771 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.777+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.774 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.779+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.774 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.779+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.775 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.779+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.775 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.780+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.782+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.782+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.782+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.783+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.783+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.777 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.783+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.777 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.783+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.777 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.785+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.778 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.785+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.779 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.785+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.782 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.787+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.782 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.787+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.782 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.788+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.783 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.790+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.783 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.790+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.787 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.792+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.787 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.792+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.787 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.792+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.788 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.793+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.788 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.793+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.795+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.795+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.795+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.796+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.791 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.796+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.797+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.797+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.798+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.800+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.792 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.801+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.793 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.802+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.793 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.802+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.794 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.803+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.798 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.804+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.798 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.805+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.798 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.805+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.798 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.806+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.799 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.807+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.799 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.807+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.799 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.808+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.800 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.809+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.801 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.810+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.801 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.810+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.802 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.811+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.802 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.811+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.802 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.811+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.803 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.811+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.803 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.811+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.805 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.812+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.806 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.813+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.806 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.813+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.806 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.814+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.806 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.815+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.808 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.816+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.808 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.817+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.808 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.817+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.810 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.818+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.810 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.818+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.811 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.818+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.811 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.819+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.811 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.819+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.811 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.820+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.812 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.820+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.813 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.821+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.813 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.822+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.814 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.822+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.814 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.822+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.816 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.824+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.816 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.824+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.816 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.824+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.817 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.826+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.817 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.826+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.818 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.827+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.820 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.828+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.820 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.828+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.820 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.828+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.820 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.829+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.821 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.830+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.821 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.830+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.821 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.830+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.822 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.831+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.822 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.831+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.827 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.832+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.829 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.833+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.829 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.833+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.829 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.833+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.834+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.834+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.835+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.835+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.835+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.835+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.830 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.836+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.831 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.837+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.831 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.837+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.831 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.838+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.831 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.838+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.832 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.839+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.832 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.839+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.832 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.839+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.832 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.840+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.833 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.841+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.833 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.841+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.833 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.842+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.834 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.842+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.834 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.842+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.835 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.844+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.835 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.844+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.836 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.845+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.837 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.846+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.837 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.846+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.846+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.846+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.846+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.842 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.847+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.842 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.848+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.842 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.849+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.842 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.850+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.843 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.850+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.843 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.850+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.843 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.850+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.843 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.851+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.844 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.852+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.846 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.852+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.850 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.853+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.850 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.855+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.850 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.857+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.850 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.857+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.851 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.858+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.859+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.860+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.860+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.852 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.860+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.853 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.861+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.853 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.861+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.853 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.862+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.853 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.862+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.854 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.863+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.854 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.863+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.855 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.863+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.856 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.865+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.856 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.865+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.856 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.865+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.860 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.869+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.861 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.869+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.861 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.869+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.862 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.870+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.864 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.873+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.864 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.873+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.873+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.875+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.875+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.876+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.868 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.876+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.869 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.876+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.869 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.876+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.869 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.877+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.871 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.877+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.872 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.879+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.872 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.879+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.872 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.880+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.873 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.880+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.873 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.881+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.875 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.884+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.877 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.886+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.878 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.886+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.878 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.887+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.879 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.887+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.879 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.887+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.879 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.887+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.880 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.889+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.881 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.890+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.882 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.890+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.882 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.891+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.883 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.891+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.883 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.891+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.884 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.893+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.888 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.894+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.888 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.894+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.888 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.894+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.889 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.895+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.889 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.895+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.890 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.896+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.890 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.896+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.890 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.896+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.890 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.897+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.891 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.898+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.891 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.899+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.897 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.899+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.897 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.900+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.898 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.901+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.899 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.901+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.899 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.901+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.899 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.902+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.900 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.902+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.900 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.903+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.900 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.903+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.900 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.906+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.900 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.906+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.901 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.907+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.901 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.908+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.901 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.909+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.902 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.909+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.902 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.910+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.902 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.910+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.903 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.911+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.903 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.912+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.903 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.912+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.904 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.913+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.905 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.913+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.905 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.913+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.907 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.916+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.907 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.916+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.908 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.916+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.908 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.916+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.908 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.917+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.909 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.917+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.913 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.919+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.913 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.919+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.913 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.920+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.914 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.920+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.914 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.921+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.914 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.921+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.915 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.922+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.915 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.923+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.915 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.923+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.915 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.924+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.924+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.924+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.924+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.916 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.924+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.925+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.926+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.926+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.924 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.926+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.926 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.927+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.927 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.927+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.927 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.928+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.927 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.928+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.928 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.929+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.928 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.929+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.928 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.929+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.928 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.930+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.929 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.931+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.929 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.931+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.929 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.932+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.930 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.935+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.930 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.935+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.930 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.936+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.931 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.936+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.931 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.936+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.931 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.936+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.931 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.936+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.931 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.937+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.931 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.937+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.931 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.939+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.932 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.939+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.932 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.939+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.932 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.940+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.933 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.941+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.936 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.943+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.936 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.943+0000 7f54a3aac700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.937 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.943+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.937 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.945+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.937 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.945+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.945+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.945+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.938 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.946+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.939 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.946+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.939 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.947+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.939 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.948+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.940 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.948+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.941 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.950+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.942 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.950+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.942 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.950+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.942 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.951+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.943 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.951+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.943 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.951+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.943 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.952+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.944 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.953+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.944 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.953+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.945 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.953+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.945 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.954+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.950 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.956+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.950 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.956+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.950 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.957+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.951 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.958+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.951 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.958+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.951 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.959+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.952 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.960+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.952 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.960+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.952 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.960+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.952 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.960+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.953 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.960+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.953 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.961+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.953 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.962+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.954 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.962+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.954 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.962+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.955 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.963+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.955 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.964+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.955 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.964+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.956 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.965+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.956 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.965+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.957 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.965+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.957 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.965+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.957 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.966+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.960 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.967+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.960 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.968+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.960 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.968+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.961 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.969+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.965 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.972+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.965 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.972+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.966 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.973+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.966 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.973+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.966 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.974+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.967 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.974+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.967 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.975+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.969 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.975+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.969 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.976+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.976+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.976+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.976+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.970 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.978+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.971 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.978+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.972 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.979+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.972 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.980+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.973 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.981+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.973 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.981+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.975 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.984+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.976 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.984+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.976 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.984+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.977 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.985+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.977 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.985+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.980 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.986+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.980 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.986+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.981 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.987+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.981 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.987+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.981 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.988+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.986 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.989+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.986 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.989+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.987 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.989+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.987 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.989+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.987 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.991+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.988 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.992+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.988 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.993+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.988 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.994+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.989 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.994+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.989 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.994+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.989 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.995+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.990 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.997+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.990 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.998+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.990 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.998+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.990 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.998+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.991 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.998+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.991 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.999+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.991 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.999+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.991 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:25.999+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.994 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.001+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.995 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.002+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.995 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.002+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.995 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.003+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.995 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.003+0000 7f54a32ab700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.996 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.003+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.996 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.004+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.996 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.005+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.997 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.006+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.997 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.006+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.998 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.007+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:25.998 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.007+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.002 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.007+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.002 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.008+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.002 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.009+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.006 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.012+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.006 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.012+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.013+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.013+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.014+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.008 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.014+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.008 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.015+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.008 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.015+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.008 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.016+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.009 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.017+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.009 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.017+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.009 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.018+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.010 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.018+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.014 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.019+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.015 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.020+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.015 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.021+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.015 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.021+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.015 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.021+0000 7f549d29f700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.021+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.022+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.016 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.023+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.017 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.023+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.017 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.023+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.017 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.024+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.025+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.025+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.026+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.026+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.018 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.026+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.019 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.027+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.019 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.027+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.019 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.028+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.020 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.028+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.020 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.029+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.020 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.029+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.029+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.030+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.031+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.031+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.023 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.031+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.024 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.032+0000 7f54a4aae700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.024 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.032+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.024 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.033+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.027 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.034+0000 7f549e2a1700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.027 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.034+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.028 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.034+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.028 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.034+0000 7f549daa0700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.028 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.034+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.028 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.035+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.029 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.036+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.029 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.037+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.029 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.037+0000 7f549f2a3700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.030 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.037+0000 7f54a02a5700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.030 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.038+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.030 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.038+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.032 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.039+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.032 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.039+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.032 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.041+0000 7f54a0aa6700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.034 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.042+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.034 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.043+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.043+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.035 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.044+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.036 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.044+0000 7f54a2aaa700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.036 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.045+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.039 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.046+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.039 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.046+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.040 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.047+0000 7f549faa4700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.040 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.047+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.040 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.047+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.040 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.048+0000 7f549eaa2700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.040 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.048+0000 7f54a1aa8700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.048+0000 7f54a12a7700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.041 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.049+0000 7f54b20f7700 -1 bdev(0x5557d0102800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.042 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2020-11-16T21:31:26.042 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:31:26.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.050+0000 7f54a42ad700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:26.051+0000 7f54a22a9700 -1 bdev(0x5557d0102400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2020-11-16T21:31:26.090 INFO:tasks.ceph.osd.1:Stopped 2020-11-16T21:31:26.747 INFO:teuthology.orchestra.run.smithi120.stdout: 18 16 17248 17232 3.73929 0.980469 0.0187031 0.0160509 2020-11-16T21:31:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: 19 16 17578 17562 3.61032 1.28906 0.0105438 0.0173031 2020-11-16T21:31:28.092 INFO:tasks.ceph.osd.1:Restarting daemon 2020-11-16T21:31:28.092 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-11-16T21:31:28.095 INFO:tasks.ceph.osd.1:Started 2020-11-16T21:31:28.095 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-11-16T21:31:28.307 INFO:teuthology.orchestra.run.smithi120.stderr:no valid command found; 10 closest matches: 2020-11-16T21:31:28.307 INFO:teuthology.orchestra.run.smithi120.stderr:0 2020-11-16T21:31:28.307 INFO:teuthology.orchestra.run.smithi120.stderr:1 2020-11-16T21:31:28.307 INFO:teuthology.orchestra.run.smithi120.stderr:2 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:abort 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:assert 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:config diff 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:config diff get 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:config get 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:config help [] 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:config set ... 2020-11-16T21:31:28.308 INFO:teuthology.orchestra.run.smithi120.stderr:admin_socket: invalid command 2020-11-16T21:31:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-11-16T21:31:28.310 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2020-11-16T21:31:28.747 INFO:teuthology.orchestra.run.smithi120.stdout:2020-11-16T21:31:28.756349+0000 min lat: 0.00393231 max lat: 1.52345 avg lat: 0.0167388 2020-11-16T21:31:28.747 INFO:teuthology.orchestra.run.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T21:31:28.747 INFO:teuthology.orchestra.run.smithi120.stdout: 20 16 19126 19110 3.73213 6.04688 0.0156585 0.0167388 2020-11-16T21:31:28.767 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:28.776+0000 7fde8be22ec0 -1 Falling back to public interface 2020-11-16T21:31:29.089 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:29.098+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10033a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:29.147 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:29.156+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10033a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:29.208 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:29.217+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:29.747 INFO:teuthology.orchestra.run.smithi120.stdout: 21 16 20529 20513 3.81536 5.48047 0.0128162 0.0163727 2020-11-16T21:31:29.769 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:29.777+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:30.241 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:30.250+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10033a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:30.301 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:30.310+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:30.747 INFO:teuthology.orchestra.run.smithi120.stdout: 22 16 21912 21896 3.88748 5.40234 0.00935971 0.0160703 2020-11-16T21:31:31.048 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:31.057+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10033a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:31.262 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:31.270+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:31.262 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:31.271+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:31.376 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:31.385+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:31.414 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:31.422+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:31.747 INFO:teuthology.orchestra.run.smithi120.stdout: 23 16 23234 23218 3.94297 5.16406 0.00933128 0.0158435 2020-11-16T21:31:32.084 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:32.093+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:32.652 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:32.660+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:32.747 INFO:teuthology.orchestra.run.smithi120.stdout: 24 16 24518 24502 3.98765 5.01562 0.0110471 0.0156678 2020-11-16T21:31:33.218 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:33.226+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:33.311 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-11-16T21:31:33.522 INFO:teuthology.orchestra.run.smithi120.stderr:no valid command found; 10 closest matches: 2020-11-16T21:31:33.523 INFO:teuthology.orchestra.run.smithi120.stderr:0 2020-11-16T21:31:33.523 INFO:teuthology.orchestra.run.smithi120.stderr:1 2020-11-16T21:31:33.523 INFO:teuthology.orchestra.run.smithi120.stderr:2 2020-11-16T21:31:33.523 INFO:teuthology.orchestra.run.smithi120.stderr:abort 2020-11-16T21:31:33.523 INFO:teuthology.orchestra.run.smithi120.stderr:assert 2020-11-16T21:31:33.523 INFO:teuthology.orchestra.run.smithi120.stderr:bluefs files list 2020-11-16T21:31:33.523 INFO:teuthology.orchestra.run.smithi120.stderr:bluefs stats 2020-11-16T21:31:33.524 INFO:teuthology.orchestra.run.smithi120.stderr:bluestore bluefs available [] 2020-11-16T21:31:33.524 INFO:teuthology.orchestra.run.smithi120.stderr:config diff 2020-11-16T21:31:33.524 INFO:teuthology.orchestra.run.smithi120.stderr:config diff get 2020-11-16T21:31:33.524 INFO:teuthology.orchestra.run.smithi120.stderr:admin_socket: invalid command 2020-11-16T21:31:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-11-16T21:31:33.525 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2020-11-16T21:31:33.747 INFO:teuthology.orchestra.run.smithi120.stdout: 25 16 25780 25764 4.02531 4.92969 0.0132473 0.0155204 2020-11-16T21:31:34.182 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:34.190+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:34.241 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:34.249+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:34.740 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:34.748+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:34.747 INFO:teuthology.orchestra.run.smithi120.stdout: 26 16 27038 27022 4.05948 4.91406 0.0233046 0.0153887 2020-11-16T21:31:35.311 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:35.320+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:35.747 INFO:teuthology.orchestra.run.smithi120.stdout: 27 16 28249 28233 4.08432 4.73047 0.007278 0.015294 2020-11-16T21:31:36.747 INFO:teuthology.orchestra.run.smithi120.stdout: 28 16 29385 29369 4.09692 4.4375 0.0113084 0.015248 2020-11-16T21:31:36.880 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:36.888+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-11-16T21:31:37.708 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:37.716+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:37.747 INFO:teuthology.orchestra.run.smithi120.stdout: 29 16 30494 30478 4.10501 4.33203 0.0200805 0.01522 2020-11-16T21:31:37.788 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:37.797+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x55d3928ce4a0 legacy=0x7f6e10035e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:38.377 INFO:tasks.ceph.osd.1.smithi120.stderr:2020-11-16T21:31:38.385+0000 7fde8be22ec0 -1 osd.1 25 log_to_monitors {default=true} 2020-11-16T21:31:38.482 INFO:teuthology.orchestra.run.smithi120.stderr:2020-11-16T21:31:38.491+0000 7f6e23a05700 0 -- 172.21.15.120:0/870913019 >> v1:172.21.15.120:6804/21986 conn(0x7f6e100336a0 legacy=0x7f6e100397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-11-16T21:31:38.526 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-11-16T21:31:38.697 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2020-11-16T21:31:38.698 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2020-11-16T21:31:38.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.698 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31408 1.6 1.2a879d2e (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.698 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.613699+0000", 2020-11-16T21:31:38.698 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.092030314000000002, 2020-11-16T21:31:38.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.092051814999999995, 2020-11-16T21:31:38.699 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.699 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.699 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.699 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.699 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.699 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31408 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613699+0000", 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613699+0000", 2020-11-16T21:31:38.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999704 2020-11-16T21:31:38.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613670+0000", 2020-11-16T21:31:38.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2238999999999996e-05 2020-11-16T21:31:38.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613752+0000", 2020-11-16T21:31:38.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3499999999999999e-05 2020-11-16T21:31:38.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613765+0000", 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7819999999999999e-06 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613774+0000", 2020-11-16T21:31:38.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0603000000000001e-05 2020-11-16T21:31:38.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613855+0000", 2020-11-16T21:31:38.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7319000000000001e-05 2020-11-16T21:31:38.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op must wait for map", 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613872+0000", 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0038021579999999999 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617674+0000", 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8040000000000007e-06 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617683+0000", 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039836719999999997 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.706 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.707 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.707 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31412 1.6 1.e85e5c06 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.707 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.617853+0000", 2020-11-16T21:31:38.707 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087876659999999995, 2020-11-16T21:31:38.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087956698, 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31412 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.708 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617853+0000", 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617853+0000", 2020-11-16T21:31:38.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9959497 2020-11-16T21:31:38.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.613803+0000", 2020-11-16T21:31:38.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.004102161 2020-11-16T21:31:38.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617905+0000", 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0984e-05 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617916+0000", 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1469e-05 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617928+0000", 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7849000000000002e-05 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617975+0000", 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5669999999999995e-06 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617984+0000", 2020-11-16T21:31:38.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013093800000000001 2020-11-16T21:31:38.714 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.714 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.714 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.714 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31414 1.6 1.6ee015de (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.714 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.617976+0000", 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087753368999999998, 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087874016999999999, 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.715 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31414 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617976+0000", 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617976+0000", 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999733 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.617950+0000", 2020-11-16T21:31:38.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1062000000000005e-05 2020-11-16T21:31:38.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618021+0000", 2020-11-16T21:31:38.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1589999999999999e-06 2020-11-16T21:31:38.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618030+0000", 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0439999999999997e-06 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618034+0000", 2020-11-16T21:31:38.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6769000000000001e-05 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618061+0000", 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2480000000000001e-06 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618067+0000", 2020-11-16T21:31:38.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0789000000000001e-05 2020-11-16T21:31:38.721 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.721 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.721 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31415 1.6 1.aa4069ae (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618078+0000", 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087652074999999996, 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087810394999999999, 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.722 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.723 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.723 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31415 2020-11-16T21:31:38.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.723 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618078+0000", 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618078+0000", 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999747 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618053+0000", 2020-11-16T21:31:38.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.826e-05 2020-11-16T21:31:38.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618121+0000", 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5180000000000004e-06 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618129+0000", 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.929e-06 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618133+0000", 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5993e-05 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618159+0000", 2020-11-16T21:31:38.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1709999999999999e-06 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618165+0000", 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7732999999999995e-05 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.728 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.729 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31417 1.0 1.eed08388 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.729 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618177+0000", 2020-11-16T21:31:38.729 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087552321000000002, 2020-11-16T21:31:38.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087749258999999996, 2020-11-16T21:31:38.729 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31417 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618177+0000", 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618177+0000", 2020-11-16T21:31:38.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999743 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618152+0000", 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8208999999999997e-05 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618220+0000", 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6500000000000002e-06 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618228+0000", 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.004e-05 2020-11-16T21:31:38.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618238+0000", 2020-11-16T21:31:38.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1402000000000001e-05 2020-11-16T21:31:38.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618280+0000", 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.909e-06 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618288+0000", 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000110375 2020-11-16T21:31:38.735 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31418 1.3 1.fc028a63 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618289+0000", 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087440806999999995, 2020-11-16T21:31:38.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087674752999999994, 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31418 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.737 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618289+0000", 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618289+0000", 2020-11-16T21:31:38.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999747 2020-11-16T21:31:38.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618264+0000", 2020-11-16T21:31:38.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9734999999999996e-05 2020-11-16T21:31:38.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618334+0000", 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7320000000000004e-06 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618342+0000", 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8089999999999999e-06 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618346+0000", 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3917000000000001e-05 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618380+0000", 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2409999999999998e-06 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618387+0000", 2020-11-16T21:31:38.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8288000000000005e-05 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31419 1.4 1.5313dad4 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618390+0000", 2020-11-16T21:31:38.743 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087339579000000001, 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087610309999999997, 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31419 2020-11-16T21:31:38.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618390+0000", 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618390+0000", 2020-11-16T21:31:38.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999752 2020-11-16T21:31:38.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618366+0000", 2020-11-16T21:31:38.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6667000000000004e-05 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618432+0000", 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7479999999999999e-06 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618441+0000", 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8079999999999998e-06 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618445+0000", 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00036583399999999999 2020-11-16T21:31:38.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618811+0000", 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1109999999999997e-06 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618820+0000", 2020-11-16T21:31:38.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00042956299999999999 2020-11-16T21:31:38.750 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.750 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.750 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.750 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31420 1.7 1.2947197 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618489+0000", 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087240689999999996, 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087549091999999995, 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31420 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618489+0000", 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618489+0000", 2020-11-16T21:31:38.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999757 2020-11-16T21:31:38.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618465+0000", 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6428999999999997e-05 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618531+0000", 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5350000000000004e-06 2020-11-16T21:31:38.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618540+0000", 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7189999999999999e-06 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618544+0000", 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4388000000000003e-05 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618588+0000", 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6470000000000004e-06 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618597+0000", 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107551 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.758 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31423 1.2 1.c52f9ee2 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.758 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618593+0000", 2020-11-16T21:31:38.758 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087137106000000006, 2020-11-16T21:31:38.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.08748264, 2020-11-16T21:31:38.758 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.758 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.758 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31423 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618593+0000", 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618593+0000", 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999738 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618567+0000", 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1330999999999999e-05 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618638+0000", 2020-11-16T21:31:38.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6579999999999997e-06 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618648+0000", 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2119999999999997e-06 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618652+0000", 2020-11-16T21:31:38.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0503999999999997e-05 2020-11-16T21:31:38.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618692+0000", 2020-11-16T21:31:38.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9280000000000005e-06 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618702+0000", 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000109523 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.764 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31424 1.4 1.6b51a254 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618704+0000", 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.087025789000000006, 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087408122000000005, 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.765 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31424 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618704+0000", 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618704+0000", 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999676 2020-11-16T21:31:38.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618672+0000", 2020-11-16T21:31:38.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1984999999999994e-05 2020-11-16T21:31:38.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618764+0000", 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0729999999999994e-06 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618773+0000", 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0280000000000002e-06 2020-11-16T21:31:38.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618777+0000", 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3367999999999994e-05 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618840+0000", 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.395e-06 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618846+0000", 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014164700000000001 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.771 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31425 1.4 1.fc16c2bc (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618821+0000", 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.086909037999999994, 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087328085, 2020-11-16T21:31:38.772 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.773 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.773 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31425 2020-11-16T21:31:38.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.773 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618821+0000", 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618821+0000", 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999733 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618794+0000", 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3331000000000007e-05 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618867+0000", 2020-11-16T21:31:38.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2940000000000006e-06 2020-11-16T21:31:38.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618876+0000", 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7230000000000002e-06 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618879+0000", 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5110000000000001e-05 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618915+0000", 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.571e-06 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618922+0000", 2020-11-16T21:31:38.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010141000000000001 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31426 1.2 1.85ad1b92 (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.618925+0000", 2020-11-16T21:31:38.779 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.086804538000000001, 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087262044999999996, 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31426 2020-11-16T21:31:38.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618925+0000", 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618925+0000", 2020-11-16T21:31:38.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999738 2020-11-16T21:31:38.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618899+0000", 2020-11-16T21:31:38.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0642000000000005e-05 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618980+0000", 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0604e-05 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618990+0000", 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.656e-06 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.618995+0000", 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016982100000000001 2020-11-16T21:31:38.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619165+0000", 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3139999999999993e-06 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.785 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619174+0000", 2020-11-16T21:31:38.786 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00024884899999999999 2020-11-16T21:31:38.786 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.786 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.786 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.786 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.786 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31427 1.7 1.cab6253f (undecoded) ondisk+retry+write+known_if_redirected e28)", 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.619043+0000", 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.086686291999999998, 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087181883000000002, 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.787 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31427 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619043+0000", 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619043+0000", 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999733 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619017+0000", 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9220000000000004e-05 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619096+0000", 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0156000000000001e-05 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619106+0000", 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2710000000000003e-06 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619110+0000", 2020-11-16T21:31:38.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9140000000000001e-05 2020-11-16T21:31:38.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.792 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.792 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.792 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619149+0000", 2020-11-16T21:31:38.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8679999999999994e-06 2020-11-16T21:31:38.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.792 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619159+0000", 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000115816 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.793 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31428 1.4 1.2787b30c (undecoded) ondisk+write+known_if_redirected e28)", 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.619161+0000", 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.086569074999999995, 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087101856000000005, 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.794 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31428 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619161+0000", 2020-11-16T21:31:38.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619161+0000", 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999704 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619131+0000", 2020-11-16T21:31:38.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0033999999999999e-05 2020-11-16T21:31:38.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619221+0000", 2020-11-16T21:31:38.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3859999999999996e-06 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619230+0000", 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5659999999999997e-06 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619236+0000", 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6672999999999999e-05 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619283+0000", 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0129999999999993e-06 2020-11-16T21:31:38.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.800 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619291+0000", 2020-11-16T21:31:38.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012997400000000001 2020-11-16T21:31:38.800 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.800 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.800 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31429 1.2 1.f43041e2 (undecoded) ondisk+write+known_if_redirected e28)", 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.619283+0000", 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.086446506000000006, 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.087032704000000002, 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.801 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31429 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619283+0000", 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619283+0000", 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999738 2020-11-16T21:31:38.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619257+0000", 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4451999999999998e-05 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619332+0000", 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0281e-05 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619342+0000", 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7809999999999996e-06 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.806 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.806 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619347+0000", 2020-11-16T21:31:38.806 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8495000000000002e-05 2020-11-16T21:31:38.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.806 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.806 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.806 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619405+0000", 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2179999999999999e-06 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619413+0000", 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013019299999999999 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.807 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.4617.0:31430 1.2 1.2e631baa (undecoded) ondisk+write+known_if_redirected e28)", 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2020-11-16T21:31:38.619422+0000", 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.086307367999999995, 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.086930454000000004, 2020-11-16T21:31:38.808 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "delayed", 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.4617", 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "v1:172.21.15.120:0/870913019", 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 31430 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2020-11-16T21:31:38.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2020-11-16T21:31:38.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619422+0000", 2020-11-16T21:31:38.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2020-11-16T21:31:38.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2020-11-16T21:31:38.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619422+0000", 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4294967295.9999719 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619394+0000", 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9637999999999999e-05 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619474+0000", 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0016e-05 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619484+0000", 2020-11-16T21:31:38.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8559999999999998e-06 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619489+0000", 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1712999999999999e-05 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.813 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2020-11-16T21:31:38.814 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619531+0000", 2020-11-16T21:31:38.814 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2120000000000003e-06 2020-11-16T21:31:38.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2020-11-16T21:31:38.814 INFO:teuthology.orchestra.run.smithi120.stdout: { 2020-11-16T21:31:38.814 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2020-11-16T21:31:38.814 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2020-11-16T21:31:38.619539+0000", 2020-11-16T21:31:38.814 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000116521 2020-11-16T21:31:38.815 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.815 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2020-11-16T21:31:38.815 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.815 INFO:teuthology.orchestra.run.smithi120.stdout: } 2020-11-16T21:31:38.815 INFO:teuthology.orchestra.run.smithi120.stdout: ], 2020-11-16T21:31:38.815 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 16 2020-11-16T21:31:38.815 INFO:teuthology.orchestra.run.smithi120.stdout:} 2020-11-16T21:31:38.816 INFO:teuthology.orchestra.run.smithi120.stdout: 30 16 31430 31414 4.09004 3.65625 0.0164438 0.0151994 2020-11-16T21:31:39.748 INFO:teuthology.orchestra.run.smithi120.stdout: 31 16 31430 31414 3.9581 0 - 0.0151994 2020-11-16T21:31:40.748 INFO:teuthology.orchestra.run.smithi120.stdout: 32 16 31430 31414 3.8344 0 - 0.0151994 2020-11-16T21:31:41.748 INFO:teuthology.orchestra.run.smithi120.stdout:Total time run: 32.8558 2020-11-16T21:31:41.748 INFO:teuthology.orchestra.run.smithi120.stdout:Total writes made: 31430 2020-11-16T21:31:41.748 INFO:teuthology.orchestra.run.smithi120.stdout:Write size: 4096 2020-11-16T21:31:41.749 INFO:teuthology.orchestra.run.smithi120.stdout:Object size: 4096 2020-11-16T21:31:41.749 INFO:teuthology.orchestra.run.smithi120.stdout:Bandwidth (MB/sec): 3.73674 2020-11-16T21:31:41.749 INFO:teuthology.orchestra.run.smithi120.stdout:Stddev Bandwidth: 1.43006 2020-11-16T21:31:41.749 INFO:teuthology.orchestra.run.smithi120.stdout:Max bandwidth (MB/sec): 6.04688 2020-11-16T21:31:41.749 INFO:teuthology.orchestra.run.smithi120.stdout:Min bandwidth (MB/sec): 0 2020-11-16T21:31:41.749 INFO:teuthology.orchestra.run.smithi120.stdout:Average IOPS: 956 2020-11-16T21:31:41.749 INFO:teuthology.orchestra.run.smithi120.stdout:Stddev IOPS: 366.095 2020-11-16T21:31:41.750 INFO:teuthology.orchestra.run.smithi120.stdout:Max IOPS: 1548 2020-11-16T21:31:41.750 INFO:teuthology.orchestra.run.smithi120.stdout:Min IOPS: 0 2020-11-16T21:31:41.750 INFO:teuthology.orchestra.run.smithi120.stdout:Average Latency(s): 0.0167129 2020-11-16T21:31:41.750 INFO:teuthology.orchestra.run.smithi120.stdout:Stddev Latency(s): 0.0752606 2020-11-16T21:31:41.750 INFO:teuthology.orchestra.run.smithi120.stdout:Max latency(s): 3.00213 2020-11-16T21:31:41.750 INFO:teuthology.orchestra.run.smithi120.stdout:Min latency(s): 0.00393231 2020-11-16T21:31:41.755 INFO:tasks.osd_backfill:err is 0 2020-11-16T21:31:41.756 INFO:tasks.osd_backfill.ceph_manager:waiting for osd.1 to be up 2020-11-16T21:31:41.756 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:31:42.084 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:42.085 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":30,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:31:40.385839+0000","last_up_change":"2020-11-16T21:31:38.603949+0000","last_in_change":"2020-11-16T21:30:03.448934+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":14,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":21984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":21984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":21984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":21984}]},"public_addr":"172.21.15.120:6808/21984","cluster_addr":"172.21.15.120:6809/21984","heartbeat_back_addr":"172.21.15.120:6811/21984","heartbeat_front_addr":"172.21.15.120:6810/21984","state":["exists"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":28,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":27113}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":27113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":27113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":27113}]},"public_addr":"172.21.15.120:6800/27113","cluster_addr":"172.21.15.120:6801/27113","heartbeat_back_addr":"172.21.15.120:6803/27113","heartbeat_front_addr":"172.21.15.120:6802/27113","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":65536,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":14},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[2]},{"pgid":"1.1","osds":[2]},{"pgid":"1.2","osds":[2]},{"pgid":"1.3","osds":[2]},{"pgid":"1.4","osds":[2]},{"pgid":"1.5","osds":[2]},{"pgid":"1.6","osds":[2]},{"pgid":"1.7","osds":[2]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:31:42.098 INFO:tasks.osd_backfill.ceph_manager:osd.1 is up 2020-11-16T21:31:42.098 INFO:tasks.osd_backfill.ceph_manager:waiting for osd.2 to be up 2020-11-16T21:31:42.098 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:31:42.448 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:42.449 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":30,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:31:40.385839+0000","last_up_change":"2020-11-16T21:31:38.603949+0000","last_in_change":"2020-11-16T21:30:03.448934+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":14,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":21984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":21984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":21984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":21984}]},"public_addr":"172.21.15.120:6808/21984","cluster_addr":"172.21.15.120:6809/21984","heartbeat_back_addr":"172.21.15.120:6811/21984","heartbeat_front_addr":"172.21.15.120:6810/21984","state":["exists"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":28,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":27113}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":27113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":27113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":27113}]},"public_addr":"172.21.15.120:6800/27113","cluster_addr":"172.21.15.120:6801/27113","heartbeat_back_addr":"172.21.15.120:6803/27113","heartbeat_front_addr":"172.21.15.120:6802/27113","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":65536,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":14},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[2]},{"pgid":"1.1","osds":[2]},{"pgid":"1.2","osds":[2]},{"pgid":"1.3","osds":[2]},{"pgid":"1.4","osds":[2]},{"pgid":"1.5","osds":[2]},{"pgid":"1.6","osds":[2]},{"pgid":"1.7","osds":[2]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:31:42.463 INFO:tasks.osd_backfill.ceph_manager:osd.2 is up 2020-11-16T21:31:42.463 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-16T21:31:42.635 INFO:teuthology.orchestra.run.smithi120.stdout:120259084290 2020-11-16T21:31:42.635 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-16T21:31:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:34359738400 2020-11-16T21:31:42.808 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:31:43.137 INFO:teuthology.orchestra.run.smithi120.stdout:1 2020-11-16T21:31:43.152 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084290 got 1 for osd.1 2020-11-16T21:31:44.153 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:31:44.496 INFO:teuthology.orchestra.run.smithi120.stdout:120259084291 2020-11-16T21:31:44.510 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084290 got 120259084291 for osd.1 2020-11-16T21:31:44.510 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-16T21:31:44.829 INFO:teuthology.orchestra.run.smithi120.stdout:34359738400 2020-11-16T21:31:44.842 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738400 got 34359738400 for osd.2 2020-11-16T21:31:44.843 INFO:tasks.osd_backfill.ceph_manager:waiting for recovery to complete 2020-11-16T21:31:44.843 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:45.176 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:45.178 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:45.190 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2020-11-16T21:31:44.066074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17474,"num_bytes_recovered":71569432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":515348,"kb_used_data":373124,"kb_used_omap":0,"kb_used_meta":142207,"kb_avail":188228332,"statfs":{"total":193273528320,"available":192745811968,"internally_reserved":0,"allocated":382078976,"data_stored":381701788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":145620028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11411456,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":11145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001126"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6988","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.594884+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.594884+0000","last_peered":"2020-11-16T21:31:41.594884+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.594884+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12620","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.572847+0000","last_change":"2020-11-16T21:31:41.575455+0000","last_active":"2020-11-16T21:31:41.572847+0000","last_peered":"2020-11-16T21:31:41.572847+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:41.572847+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592605+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.592605+0000","last_peered":"2020-11-16T21:31:41.592605+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.592605+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12753","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592495+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.592495+0000","last_peered":"2020-11-16T21:31:41.592495+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.592495+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12706","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.607532+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.607532+0000","last_peered":"2020-11-16T21:31:41.607532+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.607532+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"7081","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:42.803005+0000","last_change":"2020-11-16T21:31:41.553637+0000","last_active":"2020-11-16T21:31:42.803005+0000","last_peered":"2020-11-16T21:31:42.803005+0000","last_clean":"2020-11-16T21:31:26.009911+0000","last_became_active":"2020-11-16T21:31:41.545520+0000","last_became_peered":"2020-11-16T21:31:41.545520+0000","last_unstale":"2020-11-16T21:31:42.803005+0000","last_undegraded":"2020-11-16T21:31:40.437939+0000","last_fullsized":"2020-11-16T21:31:40.395797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4919}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12702","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.595442+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.595442+0000","last_peered":"2020-11-16T21:31:41.595442+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.595442+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12252","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.583332+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.583332+0000","last_peered":"2020-11-16T21:31:41.583332+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.583332+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17474,"num_bytes_recovered":71569432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450338816,"data_stored":450330672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.41699999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208784,"kb_used_data":155272,"kb_used_omap":0,"kb_used_meta":53503,"kb_avail":94163056,"statfs":{"total":96636764160,"available":96422969344,"internally_reserved":0,"allocated":158998528,"data_stored":158809934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":54787132},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:31:45.191 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:45.511 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:45.513 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:45.524 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2020-11-16T21:31:44.066074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17474,"num_bytes_recovered":71569432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":515348,"kb_used_data":373124,"kb_used_omap":0,"kb_used_meta":142207,"kb_avail":188228332,"statfs":{"total":193273528320,"available":192745811968,"internally_reserved":0,"allocated":382078976,"data_stored":381701788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":145620028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11411456,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":11145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001126"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6988","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.594884+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.594884+0000","last_peered":"2020-11-16T21:31:41.594884+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.594884+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12620","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.572847+0000","last_change":"2020-11-16T21:31:41.575455+0000","last_active":"2020-11-16T21:31:41.572847+0000","last_peered":"2020-11-16T21:31:41.572847+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:41.572847+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592605+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.592605+0000","last_peered":"2020-11-16T21:31:41.592605+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.592605+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12753","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592495+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.592495+0000","last_peered":"2020-11-16T21:31:41.592495+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.592495+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12706","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.607532+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.607532+0000","last_peered":"2020-11-16T21:31:41.607532+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.607532+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"7081","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:42.803005+0000","last_change":"2020-11-16T21:31:41.553637+0000","last_active":"2020-11-16T21:31:42.803005+0000","last_peered":"2020-11-16T21:31:42.803005+0000","last_clean":"2020-11-16T21:31:26.009911+0000","last_became_active":"2020-11-16T21:31:41.545520+0000","last_became_peered":"2020-11-16T21:31:41.545520+0000","last_unstale":"2020-11-16T21:31:42.803005+0000","last_undegraded":"2020-11-16T21:31:40.437939+0000","last_fullsized":"2020-11-16T21:31:40.395797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4919}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12702","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.595442+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.595442+0000","last_peered":"2020-11-16T21:31:41.595442+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.595442+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12252","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.583332+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.583332+0000","last_peered":"2020-11-16T21:31:41.583332+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.583332+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17474,"num_bytes_recovered":71569432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450338816,"data_stored":450330672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.41699999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208784,"kb_used_data":155272,"kb_used_omap":0,"kb_used_meta":53503,"kb_avail":94163056,"statfs":{"total":96636764160,"available":96422969344,"internally_reserved":0,"allocated":158998528,"data_stored":158809934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":54787132},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:31:45.525 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:45.870 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:45.872 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:45.884 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2020-11-16T21:31:44.066074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17474,"num_bytes_recovered":71569432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":515348,"kb_used_data":373124,"kb_used_omap":0,"kb_used_meta":142207,"kb_avail":188228332,"statfs":{"total":193273528320,"available":192745811968,"internally_reserved":0,"allocated":382078976,"data_stored":381701788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":145620028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11411456,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":11145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001126"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6988","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.594884+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.594884+0000","last_peered":"2020-11-16T21:31:41.594884+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.594884+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12620","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.572847+0000","last_change":"2020-11-16T21:31:41.575455+0000","last_active":"2020-11-16T21:31:41.572847+0000","last_peered":"2020-11-16T21:31:41.572847+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:41.572847+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592605+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.592605+0000","last_peered":"2020-11-16T21:31:41.592605+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.592605+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12753","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592495+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.592495+0000","last_peered":"2020-11-16T21:31:41.592495+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.592495+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12706","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.607532+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.607532+0000","last_peered":"2020-11-16T21:31:41.607532+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.607532+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"7081","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:42.803005+0000","last_change":"2020-11-16T21:31:41.553637+0000","last_active":"2020-11-16T21:31:42.803005+0000","last_peered":"2020-11-16T21:31:42.803005+0000","last_clean":"2020-11-16T21:31:26.009911+0000","last_became_active":"2020-11-16T21:31:41.545520+0000","last_became_peered":"2020-11-16T21:31:41.545520+0000","last_unstale":"2020-11-16T21:31:42.803005+0000","last_undegraded":"2020-11-16T21:31:40.437939+0000","last_fullsized":"2020-11-16T21:31:40.395797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4919}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12702","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.595442+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.595442+0000","last_peered":"2020-11-16T21:31:41.595442+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.595442+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12252","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.583332+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.583332+0000","last_peered":"2020-11-16T21:31:41.583332+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.583332+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17474,"num_bytes_recovered":71569432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450338816,"data_stored":450330672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.41699999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208784,"kb_used_data":155272,"kb_used_omap":0,"kb_used_meta":53503,"kb_avail":94163056,"statfs":{"total":96636764160,"available":96422969344,"internally_reserved":0,"allocated":158998528,"data_stored":158809934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":54787132},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:31:48.886 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:49.217 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:49.219 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:49.230 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2020-11-16T21:31:48.066724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18122,"num_bytes_recovered":74223640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":515348,"kb_used_data":373124,"kb_used_omap":0,"kb_used_meta":142207,"kb_avail":188228332,"statfs":{"total":193273528320,"available":192745811968,"internally_reserved":0,"allocated":382078976,"data_stored":381701788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":145620028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11411456,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-36529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":11145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":3452928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.684113"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6988","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.594884+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.594884+0000","last_peered":"2020-11-16T21:31:41.594884+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.594884+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12620","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.572847+0000","last_change":"2020-11-16T21:31:41.575455+0000","last_active":"2020-11-16T21:31:41.572847+0000","last_peered":"2020-11-16T21:31:41.572847+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:41.572847+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592605+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.592605+0000","last_peered":"2020-11-16T21:31:41.592605+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.592605+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12753","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592495+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.592495+0000","last_peered":"2020-11-16T21:31:41.592495+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.592495+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12706","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.607532+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.607532+0000","last_peered":"2020-11-16T21:31:41.607532+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.607532+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"8377","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:46.115287+0000","last_change":"2020-11-16T21:31:41.553637+0000","last_active":"2020-11-16T21:31:46.115287+0000","last_peered":"2020-11-16T21:31:46.115287+0000","last_clean":"2020-11-16T21:31:26.009911+0000","last_became_active":"2020-11-16T21:31:41.545520+0000","last_became_peered":"2020-11-16T21:31:41.545520+0000","last_unstale":"2020-11-16T21:31:46.115287+0000","last_undegraded":"2020-11-16T21:31:40.437939+0000","last_fullsized":"2020-11-16T21:31:40.395797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":3452928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5567}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12702","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.595442+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.595442+0000","last_peered":"2020-11-16T21:31:41.595442+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.595442+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12252","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.583332+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.583332+0000","last_peered":"2020-11-16T21:31:41.583332+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.583332+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18122,"num_bytes_recovered":74223640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450338816,"data_stored":450330672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.315}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208784,"kb_used_data":155272,"kb_used_omap":0,"kb_used_meta":53503,"kb_avail":94163056,"statfs":{"total":96636764160,"available":96422969344,"internally_reserved":0,"allocated":158998528,"data_stored":158809934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":54787132},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:31:49.231 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:49.573 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:49.575 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:49.586 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2020-11-16T21:31:48.066724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18122,"num_bytes_recovered":74223640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":515348,"kb_used_data":373124,"kb_used_omap":0,"kb_used_meta":142207,"kb_avail":188228332,"statfs":{"total":193273528320,"available":192745811968,"internally_reserved":0,"allocated":382078976,"data_stored":381701788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":145620028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11411456,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-36529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":11145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":3452928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.684113"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6988","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.594884+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.594884+0000","last_peered":"2020-11-16T21:31:41.594884+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.594884+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12620","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.572847+0000","last_change":"2020-11-16T21:31:41.575455+0000","last_active":"2020-11-16T21:31:41.572847+0000","last_peered":"2020-11-16T21:31:41.572847+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:41.572847+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592605+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.592605+0000","last_peered":"2020-11-16T21:31:41.592605+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.592605+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12753","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.592495+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.592495+0000","last_peered":"2020-11-16T21:31:41.592495+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.592495+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12706","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.607532+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.607532+0000","last_peered":"2020-11-16T21:31:41.607532+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.607532+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"8377","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:46.115287+0000","last_change":"2020-11-16T21:31:41.553637+0000","last_active":"2020-11-16T21:31:46.115287+0000","last_peered":"2020-11-16T21:31:46.115287+0000","last_clean":"2020-11-16T21:31:26.009911+0000","last_became_active":"2020-11-16T21:31:41.545520+0000","last_became_peered":"2020-11-16T21:31:41.545520+0000","last_unstale":"2020-11-16T21:31:46.115287+0000","last_undegraded":"2020-11-16T21:31:40.437939+0000","last_fullsized":"2020-11-16T21:31:40.395797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":3452928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5567}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12702","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.595442+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.595442+0000","last_peered":"2020-11-16T21:31:41.595442+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.595442+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12252","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.583332+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.583332+0000","last_peered":"2020-11-16T21:31:41.583332+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.583332+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18122,"num_bytes_recovered":74223640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450338816,"data_stored":450330672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.315}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208784,"kb_used_data":155272,"kb_used_omap":0,"kb_used_meta":53503,"kb_avail":94163056,"statfs":{"total":96636764160,"available":96422969344,"internally_reserved":0,"allocated":158998528,"data_stored":158809934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":54787132},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]}]}} 2020-11-16T21:31:52.588 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:52.909 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:52.911 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:52.923 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2020-11-16T21:31:52.067456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18440,"num_bytes_recovered":75526168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":520380,"kb_used_data":376684,"kb_used_omap":0,"kb_used_meta":143679,"kb_avail":188223300,"statfs":{"total":193273528320,"available":192740659200,"internally_reserved":0,"allocated":385724416,"data_stored":385347228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":147127356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001381"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6989","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.596420+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.596420+0000","last_peered":"2020-11-16T21:31:41.596420+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.596420+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"13258","reported_epoch":"31","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:51.112795+0000","last_change":"2020-11-16T21:31:49.789326+0000","last_active":"2020-11-16T21:31:51.112795+0000","last_peered":"2020-11-16T21:31:51.112795+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:51.112795+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":13135872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12637","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593693+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.593693+0000","last_peered":"2020-11-16T21:31:41.593693+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.593693+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12754","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593826+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.593826+0000","last_peered":"2020-11-16T21:31:41.593826+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.593826+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12707","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.610470+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.610470+0000","last_peered":"2020-11-16T21:31:41.610470+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.610470+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"8377","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:46.115287+0000","last_change":"2020-11-16T21:31:41.553637+0000","last_active":"2020-11-16T21:31:46.115287+0000","last_peered":"2020-11-16T21:31:46.115287+0000","last_clean":"2020-11-16T21:31:26.009911+0000","last_became_active":"2020-11-16T21:31:41.545520+0000","last_became_peered":"2020-11-16T21:31:41.545520+0000","last_unstale":"2020-11-16T21:31:46.115287+0000","last_undegraded":"2020-11-16T21:31:40.437939+0000","last_fullsized":"2020-11-16T21:31:40.395797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":3452928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5567}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12703","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.600146+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.600146+0000","last_peered":"2020-11-16T21:31:41.600146+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.600146+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12253","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.762903+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.762903+0000","last_peered":"2020-11-16T21:31:41.762903+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.762903+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18440,"num_bytes_recovered":75526168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455127040,"data_stored":455118896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.32600000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":213816,"kb_used_data":158832,"kb_used_omap":0,"kb_used_meta":54975,"kb_avail":94158024,"statfs":{"total":96636764160,"available":96417816576,"internally_reserved":0,"allocated":162643968,"data_stored":162455374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":56294460},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]}]}} 2020-11-16T21:31:52.924 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:53.251 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:53.253 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:53.264 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2020-11-16T21:31:52.067456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18440,"num_bytes_recovered":75526168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":520380,"kb_used_data":376684,"kb_used_omap":0,"kb_used_meta":143679,"kb_avail":188223300,"statfs":{"total":193273528320,"available":192740659200,"internally_reserved":0,"allocated":385724416,"data_stored":385347228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":147127356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001381"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6989","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.596420+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.596420+0000","last_peered":"2020-11-16T21:31:41.596420+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.596420+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"13258","reported_epoch":"31","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:51.112795+0000","last_change":"2020-11-16T21:31:49.789326+0000","last_active":"2020-11-16T21:31:51.112795+0000","last_peered":"2020-11-16T21:31:51.112795+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:51.112795+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":13135872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12637","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593693+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.593693+0000","last_peered":"2020-11-16T21:31:41.593693+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.593693+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12754","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593826+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.593826+0000","last_peered":"2020-11-16T21:31:41.593826+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.593826+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12707","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.610470+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.610470+0000","last_peered":"2020-11-16T21:31:41.610470+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.610470+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"8377","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:46.115287+0000","last_change":"2020-11-16T21:31:41.553637+0000","last_active":"2020-11-16T21:31:46.115287+0000","last_peered":"2020-11-16T21:31:46.115287+0000","last_clean":"2020-11-16T21:31:26.009911+0000","last_became_active":"2020-11-16T21:31:41.545520+0000","last_became_peered":"2020-11-16T21:31:41.545520+0000","last_unstale":"2020-11-16T21:31:46.115287+0000","last_undegraded":"2020-11-16T21:31:40.437939+0000","last_fullsized":"2020-11-16T21:31:40.395797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":3452928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5567}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12703","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.600146+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.600146+0000","last_peered":"2020-11-16T21:31:41.600146+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.600146+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12253","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.762903+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.762903+0000","last_peered":"2020-11-16T21:31:41.762903+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.762903+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18440,"num_bytes_recovered":75526168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455127040,"data_stored":455118896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.32600000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":213816,"kb_used_data":158832,"kb_used_omap":0,"kb_used_meta":54975,"kb_avail":94158024,"statfs":{"total":96636764160,"available":96417816576,"internally_reserved":0,"allocated":162643968,"data_stored":162455374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":56294460},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]}]}} 2020-11-16T21:31:56.265 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:56.594 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:56.595 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:56.606 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2020-11-16T21:31:56.068083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19527,"num_bytes_recovered":79978520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":521332,"kb_used_data":383140,"kb_used_omap":0,"kb_used_meta":138175,"kb_avail":188222348,"statfs":{"total":193273528320,"available":192739684352,"internally_reserved":0,"allocated":392335360,"data_stored":391946885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":141491260},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1405,"num_bytes_recovered":5754880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001358"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6989","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.596420+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.596420+0000","last_peered":"2020-11-16T21:31:41.596420+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.596420+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"13258","reported_epoch":"31","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:51.112795+0000","last_change":"2020-11-16T21:31:49.789326+0000","last_active":"2020-11-16T21:31:51.112795+0000","last_peered":"2020-11-16T21:31:51.112795+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:51.112795+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":13135872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12637","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593693+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.593693+0000","last_peered":"2020-11-16T21:31:41.593693+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.593693+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12754","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593826+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.593826+0000","last_peered":"2020-11-16T21:31:41.593826+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.593826+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12707","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.610470+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.610470+0000","last_peered":"2020-11-16T21:31:41.610470+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.610470+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6696","reported_epoch":"32","state":"active+clean","last_fresh":"2020-11-16T21:31:51.423845+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:51.423845+0000","last_peered":"2020-11-16T21:31:51.423845+0000","last_clean":"2020-11-16T21:31:51.423845+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:51.423845+0000","last_undegraded":"2020-11-16T21:31:51.423845+0000","last_fullsized":"2020-11-16T21:31:51.423845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12703","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.600146+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.600146+0000","last_peered":"2020-11-16T21:31:41.600146+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.600146+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12253","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.762903+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.762903+0000","last_peered":"2020-11-16T21:31:41.762903+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.762903+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19527,"num_bytes_recovered":79978520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":460582912,"data_stored":460574768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.32600000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":214768,"kb_used_data":165288,"kb_used_omap":0,"kb_used_meta":49471,"kb_avail":94157072,"statfs":{"total":96636764160,"available":96416841728,"internally_reserved":0,"allocated":169254912,"data_stored":169055031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":50658364},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]}]}} 2020-11-16T21:31:56.608 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:31:56.938 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:31:56.940 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:31:56.953 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2020-11-16T21:31:56.068083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19527,"num_bytes_recovered":79978520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":521332,"kb_used_data":383140,"kb_used_omap":0,"kb_used_meta":138175,"kb_avail":188222348,"statfs":{"total":193273528320,"available":192739684352,"internally_reserved":0,"allocated":392335360,"data_stored":391946885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":141491260},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1405,"num_bytes_recovered":5754880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001358"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6989","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.596420+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:41.596420+0000","last_peered":"2020-11-16T21:31:41.596420+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:41.596420+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"13258","reported_epoch":"31","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:51.112795+0000","last_change":"2020-11-16T21:31:49.789326+0000","last_active":"2020-11-16T21:31:51.112795+0000","last_peered":"2020-11-16T21:31:51.112795+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:41.566688+0000","last_became_peered":"2020-11-16T21:31:41.566688+0000","last_unstale":"2020-11-16T21:31:51.112795+0000","last_undegraded":"2020-11-16T21:31:40.441514+0000","last_fullsized":"2020-11-16T21:31:40.395962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":13135872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12637","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593693+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:41.593693+0000","last_peered":"2020-11-16T21:31:41.593693+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:41.593693+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12754","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.593826+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:41.593826+0000","last_peered":"2020-11-16T21:31:41.593826+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:41.593826+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12707","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.610470+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:41.610470+0000","last_peered":"2020-11-16T21:31:41.610470+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:41.610470+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6696","reported_epoch":"32","state":"active+clean","last_fresh":"2020-11-16T21:31:51.423845+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:51.423845+0000","last_peered":"2020-11-16T21:31:51.423845+0000","last_clean":"2020-11-16T21:31:51.423845+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:51.423845+0000","last_undegraded":"2020-11-16T21:31:51.423845+0000","last_fullsized":"2020-11-16T21:31:51.423845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12703","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.600146+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:41.600146+0000","last_peered":"2020-11-16T21:31:41.600146+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:41.600146+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12253","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:41.762903+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:41.762903+0000","last_peered":"2020-11-16T21:31:41.762903+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:41.762903+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19527,"num_bytes_recovered":79978520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":460582912,"data_stored":460574768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306564,"kb_used_data":217852,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065276,"statfs":{"total":96636764160,"available":96322842624,"internally_reserved":0,"allocated":223080448,"data_stored":222891854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.32600000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":214768,"kb_used_data":165288,"kb_used_omap":0,"kb_used_meta":49471,"kb_avail":94157072,"statfs":{"total":96636764160,"available":96416841728,"internally_reserved":0,"allocated":169254912,"data_stored":169055031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":50658364},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]}]}} 2020-11-16T21:31:59.954 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:00.270 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:00.272 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:00.284 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2020-11-16T21:32:00.068662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":78675992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":532076,"kb_used_data":387868,"kb_used_omap":0,"kb_used_meta":144191,"kb_avail":188211604,"statfs":{"total":193273528320,"available":192728682496,"internally_reserved":0,"allocated":397176832,"data_stored":396777070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":147651644},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":769,"num_bytes_recovered":3149824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001206"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6990","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.406116+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:50.406116+0000","last_peered":"2020-11-16T21:31:50.406116+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:50.406116+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12618","reported_epoch":"33","state":"peering","last_fresh":"2020-11-16T21:31:57.427179+0000","last_change":"2020-11-16T21:31:57.427179+0000","last_active":"2020-11-16T21:31:39.390094+0000","last_peered":"2020-11-16T21:31:38.613100+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:27.543910+0000","last_became_peered":"2020-11-16T21:31:27.543910+0000","last_unstale":"2020-11-16T21:31:57.427179+0000","last_undegraded":"2020-11-16T21:31:57.427179+0000","last_fullsized":"2020-11-16T21:31:57.427179+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12638","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.405040+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:50.405040+0000","last_peered":"2020-11-16T21:31:50.405040+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:50.405040+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12755","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.404938+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:50.404938+0000","last_peered":"2020-11-16T21:31:50.404938+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:50.404938+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12708","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.405103+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:50.405103+0000","last_peered":"2020-11-16T21:31:50.405103+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:50.405103+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6697","reported_epoch":"32","state":"active+clean","last_fresh":"2020-11-16T21:31:51.424211+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:51.424211+0000","last_peered":"2020-11-16T21:31:51.424211+0000","last_clean":"2020-11-16T21:31:51.424211+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:51.424211+0000","last_undegraded":"2020-11-16T21:31:51.424211+0000","last_fullsized":"2020-11-16T21:31:51.424211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12704","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.405140+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:50.405140+0000","last_peered":"2020-11-16T21:31:50.405140+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:50.405140+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12254","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.406074+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:50.406074+0000","last_peered":"2020-11-16T21:31:50.406074+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:50.406074+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":78675992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466108416,"data_stored":466100272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306580,"kb_used_data":217868,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065260,"statfs":{"total":96636764160,"available":96322826240,"internally_reserved":0,"allocated":223096832,"data_stored":222896951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.32300000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":225496,"kb_used_data":170000,"kb_used_omap":0,"kb_used_meta":55487,"kb_avail":94146344,"statfs":{"total":96636764160,"available":96405856256,"internally_reserved":0,"allocated":174080000,"data_stored":173880119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":56818748},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]}]}} 2020-11-16T21:32:00.285 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:00.612 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:00.625 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2020-11-16T21:32:00.068662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":78675992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":532076,"kb_used_data":387868,"kb_used_omap":0,"kb_used_meta":144191,"kb_avail":188211604,"statfs":{"total":193273528320,"available":192728682496,"internally_reserved":0,"allocated":397176832,"data_stored":396777070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":147651644},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":769,"num_bytes_recovered":3149824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001206"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6990","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.406116+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:50.406116+0000","last_peered":"2020-11-16T21:31:50.406116+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:50.406116+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12618","reported_epoch":"33","state":"peering","last_fresh":"2020-11-16T21:31:57.427179+0000","last_change":"2020-11-16T21:31:57.427179+0000","last_active":"2020-11-16T21:31:39.390094+0000","last_peered":"2020-11-16T21:31:38.613100+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:27.543910+0000","last_became_peered":"2020-11-16T21:31:27.543910+0000","last_unstale":"2020-11-16T21:31:57.427179+0000","last_undegraded":"2020-11-16T21:31:57.427179+0000","last_fullsized":"2020-11-16T21:31:57.427179+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12638","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.405040+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:50.405040+0000","last_peered":"2020-11-16T21:31:50.405040+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:50.405040+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12755","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.404938+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:50.404938+0000","last_peered":"2020-11-16T21:31:50.404938+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:50.404938+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12708","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.405103+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:50.405103+0000","last_peered":"2020-11-16T21:31:50.405103+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:50.405103+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6697","reported_epoch":"32","state":"active+clean","last_fresh":"2020-11-16T21:31:51.424211+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:51.424211+0000","last_peered":"2020-11-16T21:31:51.424211+0000","last_clean":"2020-11-16T21:31:51.424211+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:51.424211+0000","last_undegraded":"2020-11-16T21:31:51.424211+0000","last_fullsized":"2020-11-16T21:31:51.424211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12704","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.405140+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:50.405140+0000","last_peered":"2020-11-16T21:31:50.405140+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:50.405140+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"12254","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:50.406074+0000","last_change":"2020-11-16T21:31:41.583332+0000","last_active":"2020-11-16T21:31:50.406074+0000","last_peered":"2020-11-16T21:31:50.406074+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:31:50.406074+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":11268120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":78675992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466108416,"data_stored":466100272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306580,"kb_used_data":217868,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94065260,"statfs":{"total":96636764160,"available":96322826240,"internally_reserved":0,"allocated":223096832,"data_stored":222896951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.32300000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":225496,"kb_used_data":170000,"kb_used_omap":0,"kb_used_meta":55487,"kb_avail":94146344,"statfs":{"total":96636764160,"available":96405856256,"internally_reserved":0,"allocated":174080000,"data_stored":173880119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":56818748},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]}]}} 2020-11-16T21:32:03.627 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:03.974 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:03.976 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:03.987 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2020-11-16T21:32:02.068954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20063,"num_bytes_recovered":82173976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":532156,"kb_used_data":387884,"kb_used_omap":0,"kb_used_meta":144255,"kb_avail":188211524,"statfs":{"total":193273528320,"available":192728600576,"internally_reserved":0,"allocated":397193216,"data_stored":396782117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":147717180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001160"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6992","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431361+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:57.431361+0000","last_peered":"2020-11-16T21:31:57.431361+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:57.431361+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12618","reported_epoch":"33","state":"peering","last_fresh":"2020-11-16T21:31:57.427179+0000","last_change":"2020-11-16T21:31:57.427179+0000","last_active":"2020-11-16T21:31:39.390094+0000","last_peered":"2020-11-16T21:31:38.613100+0000","last_clean":"2020-11-16T21:31:26.042493+0000","last_became_active":"2020-11-16T21:31:27.543910+0000","last_became_peered":"2020-11-16T21:31:27.543910+0000","last_unstale":"2020-11-16T21:31:57.427179+0000","last_undegraded":"2020-11-16T21:31:57.427179+0000","last_fullsized":"2020-11-16T21:31:57.427179+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":11833344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12640","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431876+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:57.431876+0000","last_peered":"2020-11-16T21:31:57.431876+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:57.431876+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12757","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431370+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:57.431370+0000","last_peered":"2020-11-16T21:31:57.431370+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:57.431370+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12710","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431709+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:57.431709+0000","last_peered":"2020-11-16T21:31:57.431709+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:57.431709+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6697","reported_epoch":"32","state":"active+clean","last_fresh":"2020-11-16T21:31:51.424211+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:51.424211+0000","last_peered":"2020-11-16T21:31:51.424211+0000","last_clean":"2020-11-16T21:31:51.424211+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:51.424211+0000","last_undegraded":"2020-11-16T21:31:51.424211+0000","last_fullsized":"2020-11-16T21:31:51.424211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12706","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431664+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:57.431664+0000","last_peered":"2020-11-16T21:31:57.431664+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:57.431664+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"13965","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:01.115162+0000","last_change":"2020-11-16T21:31:57.087327+0000","last_active":"2020-11-16T21:32:01.115162+0000","last_peered":"2020-11-16T21:32:01.115162+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:32:01.115162+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":14766104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5532}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20063,"num_bytes_recovered":82173976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466108416,"data_stored":466100272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738404,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":217884,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":223113216,"data_stored":222901998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.33900000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":225496,"kb_used_data":170000,"kb_used_omap":0,"kb_used_meta":55487,"kb_avail":94146344,"statfs":{"total":96636764160,"available":96405856256,"internally_reserved":0,"allocated":174080000,"data_stored":173880119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":56818748},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]}]}} 2020-11-16T21:32:03.989 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:04.338 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:04.340 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:04.352 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2020-11-16T21:32:04.069255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22039,"num_bytes_recovered":90267672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":543812,"kb_used_data":393204,"kb_used_omap":0,"kb_used_meta":150591,"kb_avail":188199868,"statfs":{"total":193273528320,"available":192716664832,"internally_reserved":0,"allocated":402640896,"data_stored":402218460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154205244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2512,"num_bytes_recovered":10289152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001172"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6992","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431361+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:57.431361+0000","last_peered":"2020-11-16T21:31:57.431361+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:57.431361+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12626","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-16T21:31:58.448731+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:31:58.448731+0000","last_peered":"2020-11-16T21:31:58.448731+0000","last_clean":"2020-11-16T21:31:58.448731+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:31:58.448731+0000","last_undegraded":"2020-11-16T21:31:58.448731+0000","last_fullsized":"2020-11-16T21:31:58.448731+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12640","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431876+0000","last_change":"2020-11-16T21:31:41.593693+0000","last_active":"2020-11-16T21:31:57.431876+0000","last_peered":"2020-11-16T21:31:57.431876+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:31:57.431876+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12757","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431370+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:57.431370+0000","last_peered":"2020-11-16T21:31:57.431370+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:57.431370+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12710","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431709+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:57.431709+0000","last_peered":"2020-11-16T21:31:57.431709+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:57.431709+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6698","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-16T21:31:57.426612+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:57.426612+0000","last_peered":"2020-11-16T21:31:57.426612+0000","last_clean":"2020-11-16T21:31:57.426612+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:57.426612+0000","last_undegraded":"2020-11-16T21:31:57.426612+0000","last_fullsized":"2020-11-16T21:31:57.426612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12706","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:57.431664+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:57.431664+0000","last_peered":"2020-11-16T21:31:57.431664+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:57.431664+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"13965","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:01.115162+0000","last_change":"2020-11-16T21:31:57.087327+0000","last_active":"2020-11-16T21:32:01.115162+0000","last_peered":"2020-11-16T21:32:01.115162+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:32:01.115162+0000","last_undegraded":"2020-11-16T21:31:40.440126+0000","last_fullsized":"2020-11-16T21:31:40.395685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":14766104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5532}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22039,"num_bytes_recovered":90267672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":470974464,"data_stored":470966320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738404,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":217884,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":223113216,"data_stored":222901998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.33900000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237152,"kb_used_data":175320,"kb_used_omap":0,"kb_used_meta":61823,"kb_avail":94134688,"statfs":{"total":96636764160,"available":96393920512,"internally_reserved":0,"allocated":179527680,"data_stored":179316462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63306812},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]}]}} 2020-11-16T21:32:07.354 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:07.696 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:07.698 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:07.709 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2020-11-16T21:32:06.455351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":543812,"kb_used_data":393204,"kb_used_omap":0,"kb_used_meta":150591,"kb_avail":188199868,"statfs":{"total":193273528320,"available":192716664832,"internally_reserved":0,"allocated":402640896,"data_stored":402218460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154205244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4149,"num_bytes_recovered":16994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.020430"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6993","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440710+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:58.440710+0000","last_peered":"2020-11-16T21:31:58.440710+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:58.440710+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12626","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-16T21:31:58.448731+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:31:58.448731+0000","last_peered":"2020-11-16T21:31:58.448731+0000","last_clean":"2020-11-16T21:31:58.448731+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:31:58.448731+0000","last_undegraded":"2020-11-16T21:31:58.448731+0000","last_fullsized":"2020-11-16T21:31:58.448731+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12962","reported_epoch":"35","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:06.117112+0000","last_change":"2020-11-16T21:32:05.255148+0000","last_active":"2020-11-16T21:32:06.117112+0000","last_peered":"2020-11-16T21:32:06.117112+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:32:06.117112+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3056,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12758","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440646+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:58.440646+0000","last_peered":"2020-11-16T21:31:58.440646+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:58.440646+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12711","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440688+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:58.440688+0000","last_peered":"2020-11-16T21:31:58.440688+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:58.440688+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6698","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-16T21:31:57.426612+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:57.426612+0000","last_peered":"2020-11-16T21:31:57.426612+0000","last_clean":"2020-11-16T21:31:57.426612+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:57.426612+0000","last_undegraded":"2020-11-16T21:31:57.426612+0000","last_fullsized":"2020-11-16T21:31:57.426612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12707","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440888+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:58.440888+0000","last_peered":"2020-11-16T21:31:58.440888+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:58.440888+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16286","reported_epoch":"35","state":"peering","last_fresh":"2020-11-16T21:32:05.450499+0000","last_change":"2020-11-16T21:32:05.450933+0000","last_active":"2020-11-16T21:32:05.450499+0000","last_peered":"2020-11-16T21:32:05.254531+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:32:05.450499+0000","last_undegraded":"2020-11-16T21:32:05.450499+0000","last_fullsized":"2020-11-16T21:32:05.450499+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":470974464,"data_stored":470966320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":217884,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":223113216,"data_stored":222901998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.315},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.29299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237152,"kb_used_data":175320,"kb_used_omap":0,"kb_used_meta":61823,"kb_avail":94134688,"statfs":{"total":96636764160,"available":96393920512,"internally_reserved":0,"allocated":179527680,"data_stored":179316462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63306812},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]}]}} 2020-11-16T21:32:07.711 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:08.042 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:08.044 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:08.056 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2020-11-16T21:32:06.455351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":543812,"kb_used_data":393204,"kb_used_omap":0,"kb_used_meta":150591,"kb_avail":188199868,"statfs":{"total":193273528320,"available":192716664832,"internally_reserved":0,"allocated":402640896,"data_stored":402218460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154205244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4149,"num_bytes_recovered":16994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.020430"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6993","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440710+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:58.440710+0000","last_peered":"2020-11-16T21:31:58.440710+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:58.440710+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12626","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-16T21:31:58.448731+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:31:58.448731+0000","last_peered":"2020-11-16T21:31:58.448731+0000","last_clean":"2020-11-16T21:31:58.448731+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:31:58.448731+0000","last_undegraded":"2020-11-16T21:31:58.448731+0000","last_fullsized":"2020-11-16T21:31:58.448731+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12962","reported_epoch":"35","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:06.117112+0000","last_change":"2020-11-16T21:32:05.255148+0000","last_active":"2020-11-16T21:32:06.117112+0000","last_peered":"2020-11-16T21:32:06.117112+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:32:06.117112+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3056,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12758","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440646+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:58.440646+0000","last_peered":"2020-11-16T21:31:58.440646+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:58.440646+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12711","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440688+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:58.440688+0000","last_peered":"2020-11-16T21:31:58.440688+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:58.440688+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6698","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-16T21:31:57.426612+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:31:57.426612+0000","last_peered":"2020-11-16T21:31:57.426612+0000","last_clean":"2020-11-16T21:31:57.426612+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:31:57.426612+0000","last_undegraded":"2020-11-16T21:31:57.426612+0000","last_fullsized":"2020-11-16T21:31:57.426612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12707","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440888+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:58.440888+0000","last_peered":"2020-11-16T21:31:58.440888+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:58.440888+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16286","reported_epoch":"35","state":"peering","last_fresh":"2020-11-16T21:32:05.450499+0000","last_change":"2020-11-16T21:32:05.450933+0000","last_active":"2020-11-16T21:32:05.450499+0000","last_peered":"2020-11-16T21:32:05.254531+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:32:05.450499+0000","last_undegraded":"2020-11-16T21:32:05.450499+0000","last_fullsized":"2020-11-16T21:32:05.450499+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":470974464,"data_stored":470966320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":217884,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":223113216,"data_stored":222901998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.315},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.29299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237152,"kb_used_data":175320,"kb_used_omap":0,"kb_used_meta":61823,"kb_avail":94134688,"statfs":{"total":96636764160,"available":96393920512,"internally_reserved":0,"allocated":179527680,"data_stored":179316462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63306812},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]}]}} 2020-11-16T21:32:11.058 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:11.399 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:11.401 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:11.412 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2020-11-16T21:32:10.070169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":542928,"kb_used_data":398336,"kb_used_omap":0,"kb_used_meta":144575,"kb_avail":188200752,"statfs":{"total":193273528320,"available":192717570048,"internally_reserved":0,"allocated":407896064,"data_stored":407462241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":148044860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3295,"num_bytes_recovered":13496320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001215"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6993","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440710+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:58.440710+0000","last_peered":"2020-11-16T21:31:58.440710+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:58.440710+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12628","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.447943+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:05.447943+0000","last_peered":"2020-11-16T21:32:05.447943+0000","last_clean":"2020-11-16T21:32:05.447943+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:05.447943+0000","last_undegraded":"2020-11-16T21:32:05.447943+0000","last_fullsized":"2020-11-16T21:32:05.447943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12962","reported_epoch":"35","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:06.117112+0000","last_change":"2020-11-16T21:32:05.255148+0000","last_active":"2020-11-16T21:32:06.117112+0000","last_peered":"2020-11-16T21:32:06.117112+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:32:06.117112+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3056,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12758","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440646+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:58.440646+0000","last_peered":"2020-11-16T21:31:58.440646+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:58.440646+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12711","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440688+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:58.440688+0000","last_peered":"2020-11-16T21:31:58.440688+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:58.440688+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6700","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.448015+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:05.448015+0000","last_peered":"2020-11-16T21:32:05.448015+0000","last_clean":"2020-11-16T21:32:05.448015+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:05.448015+0000","last_undegraded":"2020-11-16T21:32:05.448015+0000","last_fullsized":"2020-11-16T21:32:05.448015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12707","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440888+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:58.440888+0000","last_peered":"2020-11-16T21:31:58.440888+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:58.440888+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16286","reported_epoch":"35","state":"peering","last_fresh":"2020-11-16T21:32:05.450499+0000","last_change":"2020-11-16T21:32:05.450933+0000","last_active":"2020-11-16T21:32:05.450499+0000","last_peered":"2020-11-16T21:32:05.254531+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:32:05.450499+0000","last_undegraded":"2020-11-16T21:32:05.450499+0000","last_fullsized":"2020-11-16T21:32:05.450499+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476209152,"data_stored":476201008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":217884,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":223113216,"data_stored":222901998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.315},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.29299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236268,"kb_used_data":180452,"kb_used_omap":0,"kb_used_meta":55807,"kb_avail":94135572,"statfs":{"total":96636764160,"available":96394825728,"internally_reserved":0,"allocated":184782848,"data_stored":184560243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":57146428},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]}]}} 2020-11-16T21:32:11.413 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:11.739 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:11.741 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:11.753 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2020-11-16T21:32:10.070169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":542928,"kb_used_data":398336,"kb_used_omap":0,"kb_used_meta":144575,"kb_avail":188200752,"statfs":{"total":193273528320,"available":192717570048,"internally_reserved":0,"allocated":407896064,"data_stored":407462241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":148044860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3295,"num_bytes_recovered":13496320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001215"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6993","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440710+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:31:58.440710+0000","last_peered":"2020-11-16T21:31:58.440710+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:31:58.440710+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12628","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.447943+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:05.447943+0000","last_peered":"2020-11-16T21:32:05.447943+0000","last_clean":"2020-11-16T21:32:05.447943+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:05.447943+0000","last_undegraded":"2020-11-16T21:32:05.447943+0000","last_fullsized":"2020-11-16T21:32:05.447943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"12962","reported_epoch":"35","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:06.117112+0000","last_change":"2020-11-16T21:32:05.255148+0000","last_active":"2020-11-16T21:32:06.117112+0000","last_peered":"2020-11-16T21:32:06.117112+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:32:06.117112+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3056,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12758","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440646+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:31:58.440646+0000","last_peered":"2020-11-16T21:31:58.440646+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:31:58.440646+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12711","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440688+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:31:58.440688+0000","last_peered":"2020-11-16T21:31:58.440688+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:31:58.440688+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6700","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.448015+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:05.448015+0000","last_peered":"2020-11-16T21:32:05.448015+0000","last_clean":"2020-11-16T21:32:05.448015+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:05.448015+0000","last_undegraded":"2020-11-16T21:32:05.448015+0000","last_fullsized":"2020-11-16T21:32:05.448015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12707","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:31:58.440888+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:31:58.440888+0000","last_peered":"2020-11-16T21:31:58.440888+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:31:58.440888+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16286","reported_epoch":"35","state":"peering","last_fresh":"2020-11-16T21:32:05.450499+0000","last_change":"2020-11-16T21:32:05.450933+0000","last_active":"2020-11-16T21:32:05.450499+0000","last_peered":"2020-11-16T21:32:05.254531+0000","last_clean":"2020-11-16T21:31:26.050031+0000","last_became_active":"2020-11-16T21:31:41.575431+0000","last_became_peered":"2020-11-16T21:31:41.575431+0000","last_unstale":"2020-11-16T21:32:05.450499+0000","last_undegraded":"2020-11-16T21:32:05.450499+0000","last_fullsized":"2020-11-16T21:32:05.450499+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23358,"num_bytes_recovered":95670296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476209152,"data_stored":476201008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":217884,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":223113216,"data_stored":222901998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.315},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.29299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236268,"kb_used_data":180452,"kb_used_omap":0,"kb_used_meta":55807,"kb_avail":94135572,"statfs":{"total":96636764160,"available":96394825728,"internally_reserved":0,"allocated":184782848,"data_stored":184560243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":57146428},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]}]}} 2020-11-16T21:32:14.755 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:15.076 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:15.078 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:15.093 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2020-11-16T21:32:14.478997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24592,"num_bytes_recovered":100724760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":554696,"kb_used_data":403768,"kb_used_omap":0,"kb_used_meta":150911,"kb_avail":188188984,"statfs":{"total":193273528320,"available":192705519616,"internally_reserved":0,"allocated":413458432,"data_stored":413013222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154532924},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":5054464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.023646"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6994","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.452838+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:05.452838+0000","last_peered":"2020-11-16T21:32:05.452838+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:05.452838+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12628","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.447943+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:05.447943+0000","last_peered":"2020-11-16T21:32:05.447943+0000","last_clean":"2020-11-16T21:32:05.447943+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:05.447943+0000","last_undegraded":"2020-11-16T21:32:05.447943+0000","last_fullsized":"2020-11-16T21:32:05.447943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"15431","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:11.115125+0000","last_change":"2020-11-16T21:32:05.255148+0000","last_active":"2020-11-16T21:32:11.115125+0000","last_peered":"2020-11-16T21:32:11.115125+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:32:11.115125+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":17571840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6250}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.451051+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:32:05.451051+0000","last_peered":"2020-11-16T21:32:05.451051+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:32:05.451051+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12712","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.451330+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:05.451330+0000","last_peered":"2020-11-16T21:32:05.451330+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:05.451330+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6700","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.448015+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:05.448015+0000","last_peered":"2020-11-16T21:32:05.448015+0000","last_clean":"2020-11-16T21:32:05.448015+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:05.448015+0000","last_undegraded":"2020-11-16T21:32:05.448015+0000","last_fullsized":"2020-11-16T21:32:05.448015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12708","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.451266+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:32:05.451266+0000","last_peered":"2020-11-16T21:32:05.451266+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:05.451266+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16294","reported_epoch":"36","state":"active+clean","last_fresh":"2020-11-16T21:32:06.471012+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:06.471012+0000","last_peered":"2020-11-16T21:32:06.471012+0000","last_clean":"2020-11-16T21:32:06.471012+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:06.471012+0000","last_undegraded":"2020-11-16T21:32:06.471012+0000","last_fullsized":"2020-11-16T21:32:06.471012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24592,"num_bytes_recovered":100724760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481755136,"data_stored":481746992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306676,"kb_used_data":217900,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065164,"statfs":{"total":96636764160,"available":96322727936,"internally_reserved":0,"allocated":223129600,"data_stored":222906995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34200000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248020,"kb_used_data":185868,"kb_used_omap":0,"kb_used_meta":62143,"kb_avail":94123820,"statfs":{"total":96636764160,"available":96382791680,"internally_reserved":0,"allocated":190328832,"data_stored":190106227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63634492},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}]}} 2020-11-16T21:32:15.094 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:15.418 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:15.421 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:15.432 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2020-11-16T21:32:14.478997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24592,"num_bytes_recovered":100724760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":554696,"kb_used_data":403768,"kb_used_omap":0,"kb_used_meta":150911,"kb_avail":188188984,"statfs":{"total":193273528320,"available":192705519616,"internally_reserved":0,"allocated":413458432,"data_stored":413013222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154532924},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1234,"num_bytes_recovered":5054464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.023646"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6994","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.452838+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:05.452838+0000","last_peered":"2020-11-16T21:32:05.452838+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:05.452838+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12628","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.447943+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:05.447943+0000","last_peered":"2020-11-16T21:32:05.447943+0000","last_clean":"2020-11-16T21:32:05.447943+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:05.447943+0000","last_undegraded":"2020-11-16T21:32:05.447943+0000","last_fullsized":"2020-11-16T21:32:05.447943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"15431","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:11.115125+0000","last_change":"2020-11-16T21:32:05.255148+0000","last_active":"2020-11-16T21:32:11.115125+0000","last_peered":"2020-11-16T21:32:11.115125+0000","last_clean":"2020-11-16T21:31:26.050098+0000","last_became_active":"2020-11-16T21:31:41.586250+0000","last_became_peered":"2020-11-16T21:31:41.586250+0000","last_unstale":"2020-11-16T21:32:11.115125+0000","last_undegraded":"2020-11-16T21:31:40.438514+0000","last_fullsized":"2020-11-16T21:31:40.395413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":17571840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6250}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.451051+0000","last_change":"2020-11-16T21:31:41.593826+0000","last_active":"2020-11-16T21:32:05.451051+0000","last_peered":"2020-11-16T21:32:05.451051+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:32:05.451051+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2926,"num_bytes_recovered":11984896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12712","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.451330+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:05.451330+0000","last_peered":"2020-11-16T21:32:05.451330+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:05.451330+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6700","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.448015+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:05.448015+0000","last_peered":"2020-11-16T21:32:05.448015+0000","last_clean":"2020-11-16T21:32:05.448015+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:05.448015+0000","last_undegraded":"2020-11-16T21:32:05.448015+0000","last_fullsized":"2020-11-16T21:32:05.448015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12708","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:05.451266+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:32:05.451266+0000","last_peered":"2020-11-16T21:32:05.451266+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:05.451266+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16294","reported_epoch":"36","state":"active+clean","last_fresh":"2020-11-16T21:32:06.471012+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:06.471012+0000","last_peered":"2020-11-16T21:32:06.471012+0000","last_clean":"2020-11-16T21:32:06.471012+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:06.471012+0000","last_undegraded":"2020-11-16T21:32:06.471012+0000","last_fullsized":"2020-11-16T21:32:06.471012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24592,"num_bytes_recovered":100724760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481755136,"data_stored":481746992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306676,"kb_used_data":217900,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94065164,"statfs":{"total":96636764160,"available":96322727936,"internally_reserved":0,"allocated":223129600,"data_stored":222906995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34200000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248020,"kb_used_data":185868,"kb_used_omap":0,"kb_used_meta":62143,"kb_avail":94123820,"statfs":{"total":96636764160,"available":96382791680,"internally_reserved":0,"allocated":190328832,"data_stored":190106227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63634492},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}]}} 2020-11-16T21:32:18.433 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:18.755 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:18.757 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:18.768 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2020-11-16T21:32:18.071440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25913,"num_bytes_recovered":106135576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":554776,"kb_used_data":403784,"kb_used_omap":0,"kb_used_meta":150975,"kb_avail":188188904,"statfs":{"total":193273528320,"available":192705437696,"internally_reserved":0,"allocated":413474816,"data_stored":413018169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154598460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2555,"num_bytes_recovered":10465280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001270"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6996","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:13.475761+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:13.475761+0000","last_peered":"2020-11-16T21:32:13.475761+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:13.475761+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12628","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.447943+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:05.447943+0000","last_peered":"2020-11-16T21:32:05.447943+0000","last_clean":"2020-11-16T21:32:05.447943+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:05.447943+0000","last_undegraded":"2020-11-16T21:32:05.447943+0000","last_fullsized":"2020-11-16T21:32:05.447943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16518","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.493184+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:14.493184+0000","last_peered":"2020-11-16T21:32:14.493184+0000","last_clean":"2020-11-16T21:32:14.493184+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:14.493184+0000","last_undegraded":"2020-11-16T21:32:14.493184+0000","last_fullsized":"2020-11-16T21:32:14.493184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"14328","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:16.112081+0000","last_change":"2020-11-16T21:32:12.777066+0000","last_active":"2020-11-16T21:32:16.112081+0000","last_peered":"2020-11-16T21:32:16.112081+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:32:16.112081+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":15192064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12714","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:13.475834+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:13.475834+0000","last_peered":"2020-11-16T21:32:13.475834+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:13.475834+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6700","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.448015+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:05.448015+0000","last_peered":"2020-11-16T21:32:05.448015+0000","last_clean":"2020-11-16T21:32:05.448015+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:05.448015+0000","last_undegraded":"2020-11-16T21:32:05.448015+0000","last_fullsized":"2020-11-16T21:32:05.448015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12710","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:13.478114+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:32:13.478114+0000","last_peered":"2020-11-16T21:32:13.478114+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:13.478114+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16296","reported_epoch":"37","state":"active+clean","last_fresh":"2020-11-16T21:32:13.474847+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:13.474847+0000","last_peered":"2020-11-16T21:32:13.474847+0000","last_clean":"2020-11-16T21:32:13.474847+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:13.474847+0000","last_undegraded":"2020-11-16T21:32:13.474847+0000","last_fullsized":"2020-11-16T21:32:13.474847+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25913,"num_bytes_recovered":106135576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481755136,"data_stored":481746992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306756,"kb_used_data":217916,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065084,"statfs":{"total":96636764160,"available":96322646016,"internally_reserved":0,"allocated":223145984,"data_stored":222911942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34000000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248020,"kb_used_data":185868,"kb_used_omap":0,"kb_used_meta":62143,"kb_avail":94123820,"statfs":{"total":96636764160,"available":96382791680,"internally_reserved":0,"allocated":190328832,"data_stored":190106227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63634492},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}]}} 2020-11-16T21:32:18.769 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:19.105 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:19.107 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:19.118 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2020-11-16T21:32:18.071440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25913,"num_bytes_recovered":106135576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":554776,"kb_used_data":403784,"kb_used_omap":0,"kb_used_meta":150975,"kb_avail":188188904,"statfs":{"total":193273528320,"available":192705437696,"internally_reserved":0,"allocated":413474816,"data_stored":413018169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154598460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2555,"num_bytes_recovered":10465280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001270"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6996","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:13.475761+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:13.475761+0000","last_peered":"2020-11-16T21:32:13.475761+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:13.475761+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12628","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.447943+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:05.447943+0000","last_peered":"2020-11-16T21:32:05.447943+0000","last_clean":"2020-11-16T21:32:05.447943+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:05.447943+0000","last_undegraded":"2020-11-16T21:32:05.447943+0000","last_fullsized":"2020-11-16T21:32:05.447943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16518","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.493184+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:14.493184+0000","last_peered":"2020-11-16T21:32:14.493184+0000","last_clean":"2020-11-16T21:32:14.493184+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:14.493184+0000","last_undegraded":"2020-11-16T21:32:14.493184+0000","last_fullsized":"2020-11-16T21:32:14.493184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"14328","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:16.112081+0000","last_change":"2020-11-16T21:32:12.777066+0000","last_active":"2020-11-16T21:32:16.112081+0000","last_peered":"2020-11-16T21:32:16.112081+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:32:16.112081+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":15192064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12714","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:13.475834+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:13.475834+0000","last_peered":"2020-11-16T21:32:13.475834+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:13.475834+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6700","reported_epoch":"35","state":"active+clean","last_fresh":"2020-11-16T21:32:05.448015+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:05.448015+0000","last_peered":"2020-11-16T21:32:05.448015+0000","last_clean":"2020-11-16T21:32:05.448015+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:05.448015+0000","last_undegraded":"2020-11-16T21:32:05.448015+0000","last_fullsized":"2020-11-16T21:32:05.448015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12710","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:13.478114+0000","last_change":"2020-11-16T21:31:41.595442+0000","last_active":"2020-11-16T21:32:13.478114+0000","last_peered":"2020-11-16T21:32:13.478114+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:13.478114+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2902,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16296","reported_epoch":"37","state":"active+clean","last_fresh":"2020-11-16T21:32:13.474847+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:13.474847+0000","last_peered":"2020-11-16T21:32:13.474847+0000","last_clean":"2020-11-16T21:32:13.474847+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:13.474847+0000","last_undegraded":"2020-11-16T21:32:13.474847+0000","last_fullsized":"2020-11-16T21:32:13.474847+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25913,"num_bytes_recovered":106135576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481755136,"data_stored":481746992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306756,"kb_used_data":217916,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065084,"statfs":{"total":96636764160,"available":96322646016,"internally_reserved":0,"allocated":223145984,"data_stored":222911942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34000000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248020,"kb_used_data":185868,"kb_used_omap":0,"kb_used_meta":62143,"kb_avail":94123820,"statfs":{"total":96636764160,"available":96382791680,"internally_reserved":0,"allocated":190328832,"data_stored":190106227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63634492},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}]}} 2020-11-16T21:32:22.120 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:22.434 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:22.436 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:22.447 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2020-11-16T21:32:22.072047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":106623000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":565832,"kb_used_data":408696,"kb_used_omap":0,"kb_used_meta":157119,"kb_avail":188177848,"statfs":{"total":193273528320,"available":192694116352,"internally_reserved":0,"allocated":418504704,"data_stored":418036620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":160889916},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.593050"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484269+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:14.484269+0000","last_peered":"2020-11-16T21:32:14.484269+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:14.484269+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12630","reported_epoch":"37","state":"active+clean","last_fresh":"2020-11-16T21:32:13.469936+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:13.469936+0000","last_peered":"2020-11-16T21:32:13.469936+0000","last_clean":"2020-11-16T21:32:13.469936+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:13.469936+0000","last_undegraded":"2020-11-16T21:32:13.469936+0000","last_fullsized":"2020-11-16T21:32:13.469936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16519","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.496187+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:14.496187+0000","last_peered":"2020-11-16T21:32:14.496187+0000","last_clean":"2020-11-16T21:32:14.496187+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:14.496187+0000","last_undegraded":"2020-11-16T21:32:14.496187+0000","last_fullsized":"2020-11-16T21:32:14.496187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"14328","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:16.112081+0000","last_change":"2020-11-16T21:32:12.777066+0000","last_active":"2020-11-16T21:32:16.112081+0000","last_peered":"2020-11-16T21:32:16.112081+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:32:16.112081+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":15192064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12715","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484246+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:14.484246+0000","last_peered":"2020-11-16T21:32:14.484246+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:14.484246+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6702","reported_epoch":"37","state":"active+clean","last_fresh":"2020-11-16T21:32:13.469857+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:13.469857+0000","last_peered":"2020-11-16T21:32:13.469857+0000","last_clean":"2020-11-16T21:32:13.469857+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:13.469857+0000","last_undegraded":"2020-11-16T21:32:13.469857+0000","last_fullsized":"2020-11-16T21:32:13.469857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12950","reported_epoch":"39","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:21.113378+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:21.113378+0000","last_peered":"2020-11-16T21:32:21.113378+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:21.113378+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12374016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16297","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.483331+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:14.483331+0000","last_peered":"2020-11-16T21:32:14.483331+0000","last_clean":"2020-11-16T21:32:14.483331+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:14.483331+0000","last_undegraded":"2020-11-16T21:32:14.483331+0000","last_fullsized":"2020-11-16T21:32:14.483331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":106623000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":486825984,"data_stored":486817840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306756,"kb_used_data":217916,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065084,"statfs":{"total":96636764160,"available":96322646016,"internally_reserved":0,"allocated":223145984,"data_stored":222911942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259076,"kb_used_data":190780,"kb_used_omap":0,"kb_used_meta":68287,"kb_avail":94112764,"statfs":{"total":96636764160,"available":96371470336,"internally_reserved":0,"allocated":195358720,"data_stored":195124678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":69925948},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]}]}} 2020-11-16T21:32:22.448 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:22.775 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:22.777 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:22.789 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2020-11-16T21:32:22.072047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":106623000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":565832,"kb_used_data":408696,"kb_used_omap":0,"kb_used_meta":157119,"kb_avail":188177848,"statfs":{"total":193273528320,"available":192694116352,"internally_reserved":0,"allocated":418504704,"data_stored":418036620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":160889916},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.593050"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484269+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:14.484269+0000","last_peered":"2020-11-16T21:32:14.484269+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:14.484269+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12630","reported_epoch":"37","state":"active+clean","last_fresh":"2020-11-16T21:32:13.469936+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:13.469936+0000","last_peered":"2020-11-16T21:32:13.469936+0000","last_clean":"2020-11-16T21:32:13.469936+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:13.469936+0000","last_undegraded":"2020-11-16T21:32:13.469936+0000","last_fullsized":"2020-11-16T21:32:13.469936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16519","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.496187+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:14.496187+0000","last_peered":"2020-11-16T21:32:14.496187+0000","last_clean":"2020-11-16T21:32:14.496187+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:14.496187+0000","last_undegraded":"2020-11-16T21:32:14.496187+0000","last_fullsized":"2020-11-16T21:32:14.496187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"14328","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:16.112081+0000","last_change":"2020-11-16T21:32:12.777066+0000","last_active":"2020-11-16T21:32:16.112081+0000","last_peered":"2020-11-16T21:32:16.112081+0000","last_clean":"2020-11-16T21:31:26.033683+0000","last_became_active":"2020-11-16T21:31:41.585603+0000","last_became_peered":"2020-11-16T21:31:41.585603+0000","last_unstale":"2020-11-16T21:32:16.112081+0000","last_undegraded":"2020-11-16T21:31:40.448000+0000","last_fullsized":"2020-11-16T21:31:40.395328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":15192064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12715","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484246+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:14.484246+0000","last_peered":"2020-11-16T21:32:14.484246+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:14.484246+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6702","reported_epoch":"37","state":"active+clean","last_fresh":"2020-11-16T21:32:13.469857+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:13.469857+0000","last_peered":"2020-11-16T21:32:13.469857+0000","last_clean":"2020-11-16T21:32:13.469857+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:13.469857+0000","last_undegraded":"2020-11-16T21:32:13.469857+0000","last_fullsized":"2020-11-16T21:32:13.469857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12950","reported_epoch":"39","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:21.113378+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:21.113378+0000","last_peered":"2020-11-16T21:32:21.113378+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:21.113378+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12374016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16297","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.483331+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:14.483331+0000","last_peered":"2020-11-16T21:32:14.483331+0000","last_clean":"2020-11-16T21:32:14.483331+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:14.483331+0000","last_undegraded":"2020-11-16T21:32:14.483331+0000","last_fullsized":"2020-11-16T21:32:14.483331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":106623000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":486825984,"data_stored":486817840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306756,"kb_used_data":217916,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065084,"statfs":{"total":96636764160,"available":96322646016,"internally_reserved":0,"allocated":223145984,"data_stored":222911942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259076,"kb_used_data":190780,"kb_used_omap":0,"kb_used_meta":68287,"kb_avail":94112764,"statfs":{"total":96636764160,"available":96371470336,"internally_reserved":0,"allocated":195358720,"data_stored":195124678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":69925948},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]}]}} 2020-11-16T21:32:25.791 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:26.129 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:26.131 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:26.142 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2020-11-16T21:32:26.072760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27228,"num_bytes_recovered":111521816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":564892,"kb_used_data":413836,"kb_used_omap":0,"kb_used_meta":151039,"kb_avail":188178788,"statfs":{"total":193273528320,"available":192695078912,"internally_reserved":0,"allocated":423768064,"data_stored":423288493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154663996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":5386240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001319"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484269+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:14.484269+0000","last_peered":"2020-11-16T21:32:14.484269+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:14.484269+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12632","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.500375+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:20.500375+0000","last_peered":"2020-11-16T21:32:20.500375+0000","last_clean":"2020-11-16T21:32:20.500375+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:20.500375+0000","last_undegraded":"2020-11-16T21:32:20.500375+0000","last_fullsized":"2020-11-16T21:32:20.500375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16519","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.496187+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:14.496187+0000","last_peered":"2020-11-16T21:32:14.496187+0000","last_clean":"2020-11-16T21:32:14.496187+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:14.496187+0000","last_undegraded":"2020-11-16T21:32:14.496187+0000","last_fullsized":"2020-11-16T21:32:14.496187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"40","state":"active+clean","last_fresh":"2020-11-16T21:32:21.511649+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:21.511649+0000","last_peered":"2020-11-16T21:32:21.511649+0000","last_clean":"2020-11-16T21:32:21.511649+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:21.511649+0000","last_undegraded":"2020-11-16T21:32:21.511649+0000","last_fullsized":"2020-11-16T21:32:21.511649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12715","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484246+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:14.484246+0000","last_peered":"2020-11-16T21:32:14.484246+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:14.484246+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6704","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.501169+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:20.501169+0000","last_peered":"2020-11-16T21:32:20.501169+0000","last_clean":"2020-11-16T21:32:20.501169+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:20.501169+0000","last_undegraded":"2020-11-16T21:32:20.501169+0000","last_fullsized":"2020-11-16T21:32:20.501169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12950","reported_epoch":"39","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:21.113378+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:21.113378+0000","last_peered":"2020-11-16T21:32:21.113378+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:21.113378+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12374016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16297","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.483331+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:14.483331+0000","last_peered":"2020-11-16T21:32:14.483331+0000","last_clean":"2020-11-16T21:32:14.483331+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:14.483331+0000","last_undegraded":"2020-11-16T21:32:14.483331+0000","last_fullsized":"2020-11-16T21:32:14.483331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27228,"num_bytes_recovered":111521816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491941888,"data_stored":491933744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306756,"kb_used_data":217916,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065084,"statfs":{"total":96636764160,"available":96322646016,"internally_reserved":0,"allocated":223145984,"data_stored":222911942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258136,"kb_used_data":195920,"kb_used_omap":0,"kb_used_meta":62207,"kb_avail":94113704,"statfs":{"total":96636764160,"available":96372432896,"internally_reserved":0,"allocated":200622080,"data_stored":200376551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63700028},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]}]}} 2020-11-16T21:32:26.144 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:26.480 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:26.482 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:26.494 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2020-11-16T21:32:26.072760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27228,"num_bytes_recovered":111521816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":564892,"kb_used_data":413836,"kb_used_omap":0,"kb_used_meta":151039,"kb_avail":188178788,"statfs":{"total":193273528320,"available":192695078912,"internally_reserved":0,"allocated":423768064,"data_stored":423288493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":154663996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":5386240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001319"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484269+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:14.484269+0000","last_peered":"2020-11-16T21:32:14.484269+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:14.484269+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12632","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.500375+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:20.500375+0000","last_peered":"2020-11-16T21:32:20.500375+0000","last_clean":"2020-11-16T21:32:20.500375+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:20.500375+0000","last_undegraded":"2020-11-16T21:32:20.500375+0000","last_fullsized":"2020-11-16T21:32:20.500375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16519","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.496187+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:14.496187+0000","last_peered":"2020-11-16T21:32:14.496187+0000","last_clean":"2020-11-16T21:32:14.496187+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:14.496187+0000","last_undegraded":"2020-11-16T21:32:14.496187+0000","last_fullsized":"2020-11-16T21:32:14.496187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"40","state":"active+clean","last_fresh":"2020-11-16T21:32:21.511649+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:21.511649+0000","last_peered":"2020-11-16T21:32:21.511649+0000","last_clean":"2020-11-16T21:32:21.511649+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:21.511649+0000","last_undegraded":"2020-11-16T21:32:21.511649+0000","last_fullsized":"2020-11-16T21:32:21.511649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12715","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:14.484246+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:14.484246+0000","last_peered":"2020-11-16T21:32:14.484246+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:14.484246+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6704","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.501169+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:20.501169+0000","last_peered":"2020-11-16T21:32:20.501169+0000","last_clean":"2020-11-16T21:32:20.501169+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:20.501169+0000","last_undegraded":"2020-11-16T21:32:20.501169+0000","last_fullsized":"2020-11-16T21:32:20.501169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12950","reported_epoch":"39","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:21.113378+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:21.113378+0000","last_peered":"2020-11-16T21:32:21.113378+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:21.113378+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12374016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16297","reported_epoch":"38","state":"active+clean","last_fresh":"2020-11-16T21:32:14.483331+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:14.483331+0000","last_peered":"2020-11-16T21:32:14.483331+0000","last_clean":"2020-11-16T21:32:14.483331+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:14.483331+0000","last_undegraded":"2020-11-16T21:32:14.483331+0000","last_fullsized":"2020-11-16T21:32:14.483331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27228,"num_bytes_recovered":111521816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491941888,"data_stored":491933744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306756,"kb_used_data":217916,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065084,"statfs":{"total":96636764160,"available":96322646016,"internally_reserved":0,"allocated":223145984,"data_stored":222911942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258136,"kb_used_data":195920,"kb_used_omap":0,"kb_used_meta":62207,"kb_avail":94113704,"statfs":{"total":96636764160,"available":96372432896,"internally_reserved":0,"allocated":200622080,"data_stored":200376551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":63700028},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]}]}} 2020-11-16T21:32:29.496 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:29.857 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:29.859 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:29.871 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2020-11-16T21:32:29.392168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28356,"num_bytes_recovered":116142104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":576444,"kb_used_data":419052,"kb_used_omap":0,"kb_used_meta":157375,"kb_avail":188167236,"statfs":{"total":193273528320,"available":192683249664,"internally_reserved":0,"allocated":429109248,"data_stored":428618190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":161152060},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2443,"num_bytes_recovered":10006528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.893453"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6998","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:20.504339+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:20.504339+0000","last_peered":"2020-11-16T21:32:20.504339+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:20.504339+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12632","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.500375+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:20.500375+0000","last_peered":"2020-11-16T21:32:20.500375+0000","last_clean":"2020-11-16T21:32:20.500375+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:20.500375+0000","last_undegraded":"2020-11-16T21:32:20.500375+0000","last_fullsized":"2020-11-16T21:32:20.500375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16520","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.503531+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:20.503531+0000","last_peered":"2020-11-16T21:32:20.503531+0000","last_clean":"2020-11-16T21:32:20.503531+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:20.503531+0000","last_undegraded":"2020-11-16T21:32:20.503531+0000","last_fullsized":"2020-11-16T21:32:20.503531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"40","state":"active+clean","last_fresh":"2020-11-16T21:32:21.511649+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:21.511649+0000","last_peered":"2020-11-16T21:32:21.511649+0000","last_clean":"2020-11-16T21:32:21.511649+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:21.511649+0000","last_undegraded":"2020-11-16T21:32:21.511649+0000","last_fullsized":"2020-11-16T21:32:21.511649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12716","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:20.504430+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:20.504430+0000","last_peered":"2020-11-16T21:32:20.504430+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:20.504430+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6704","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.501169+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:20.501169+0000","last_peered":"2020-11-16T21:32:20.501169+0000","last_clean":"2020-11-16T21:32:20.501169+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:20.501169+0000","last_undegraded":"2020-11-16T21:32:20.501169+0000","last_fullsized":"2020-11-16T21:32:20.501169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"15207","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:26.115555+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:26.115555+0000","last_peered":"2020-11-16T21:32:26.115555+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:26.115555+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4149,"num_bytes_recovered":16994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16298","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.503650+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:20.503650+0000","last_peered":"2020-11-16T21:32:20.503650+0000","last_clean":"2020-11-16T21:32:20.503650+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:20.503650+0000","last_undegraded":"2020-11-16T21:32:20.503650+0000","last_fullsized":"2020-11-16T21:32:20.503650+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28356,"num_bytes_recovered":116142104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497201152,"data_stored":497193008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306772,"kb_used_data":217932,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065068,"statfs":{"total":96636764160,"available":96322629632,"internally_reserved":0,"allocated":223162368,"data_stored":222916839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.28999999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269672,"kb_used_data":201120,"kb_used_omap":0,"kb_used_meta":68543,"kb_avail":94102168,"statfs":{"total":96636764160,"available":96360620032,"internally_reserved":0,"allocated":205946880,"data_stored":205701351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":70188092},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]}]}} 2020-11-16T21:32:29.872 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:30.210 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:30.212 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:30.223 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2020-11-16T21:32:30.073326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28356,"num_bytes_recovered":116142104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":576444,"kb_used_data":419052,"kb_used_omap":0,"kb_used_meta":157375,"kb_avail":188167236,"statfs":{"total":193273528320,"available":192683249664,"internally_reserved":0,"allocated":429109248,"data_stored":428618190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":161152060},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":9519104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.575460"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6998","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:20.504339+0000","last_change":"2020-11-16T21:31:41.596420+0000","last_active":"2020-11-16T21:32:20.504339+0000","last_peered":"2020-11-16T21:32:20.504339+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:20.504339+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12632","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.500375+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:20.500375+0000","last_peered":"2020-11-16T21:32:20.500375+0000","last_clean":"2020-11-16T21:32:20.500375+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:20.500375+0000","last_undegraded":"2020-11-16T21:32:20.500375+0000","last_fullsized":"2020-11-16T21:32:20.500375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16520","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.503531+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:20.503531+0000","last_peered":"2020-11-16T21:32:20.503531+0000","last_clean":"2020-11-16T21:32:20.503531+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:20.503531+0000","last_undegraded":"2020-11-16T21:32:20.503531+0000","last_fullsized":"2020-11-16T21:32:20.503531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"40","state":"active+clean","last_fresh":"2020-11-16T21:32:21.511649+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:21.511649+0000","last_peered":"2020-11-16T21:32:21.511649+0000","last_clean":"2020-11-16T21:32:21.511649+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:21.511649+0000","last_undegraded":"2020-11-16T21:32:21.511649+0000","last_fullsized":"2020-11-16T21:32:21.511649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12716","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:20.504430+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:20.504430+0000","last_peered":"2020-11-16T21:32:20.504430+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:20.504430+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6704","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.501169+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:20.501169+0000","last_peered":"2020-11-16T21:32:20.501169+0000","last_clean":"2020-11-16T21:32:20.501169+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:20.501169+0000","last_undegraded":"2020-11-16T21:32:20.501169+0000","last_fullsized":"2020-11-16T21:32:20.501169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"15207","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:26.115555+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:26.115555+0000","last_peered":"2020-11-16T21:32:26.115555+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:26.115555+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4149,"num_bytes_recovered":16994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16298","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.503650+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:20.503650+0000","last_peered":"2020-11-16T21:32:20.503650+0000","last_clean":"2020-11-16T21:32:20.503650+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:20.503650+0000","last_undegraded":"2020-11-16T21:32:20.503650+0000","last_fullsized":"2020-11-16T21:32:20.503650+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28356,"num_bytes_recovered":116142104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497201152,"data_stored":497193008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306772,"kb_used_data":217932,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065068,"statfs":{"total":96636764160,"available":96322629632,"internally_reserved":0,"allocated":223162368,"data_stored":222916839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.28999999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269672,"kb_used_data":201120,"kb_used_omap":0,"kb_used_meta":68543,"kb_avail":94102168,"statfs":{"total":96636764160,"available":96360620032,"internally_reserved":0,"allocated":205946880,"data_stored":205701351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":70188092},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]}]}} 2020-11-16T21:32:33.225 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:33.548 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:33.550 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2020-11-16T21:32:32.073623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28931,"num_bytes_recovered":118497304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":576452,"kb_used_data":419060,"kb_used_omap":0,"kb_used_meta":157375,"kb_avail":188167228,"statfs":{"total":193273528320,"available":192683241472,"internally_reserved":0,"allocated":429117440,"data_stored":428620624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":161152060},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1703,"num_bytes_recovered":6975488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001210"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"8151","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:31.114133+0000","last_change":"2020-11-16T21:32:28.720493+0000","last_active":"2020-11-16T21:32:31.114133+0000","last_peered":"2020-11-16T21:32:31.114133+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:31.114133+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":2355200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5469}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12632","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.500375+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:20.500375+0000","last_peered":"2020-11-16T21:32:20.500375+0000","last_clean":"2020-11-16T21:32:20.500375+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:20.500375+0000","last_undegraded":"2020-11-16T21:32:20.500375+0000","last_fullsized":"2020-11-16T21:32:20.500375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16522","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398064+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:29.398064+0000","last_peered":"2020-11-16T21:32:29.398064+0000","last_clean":"2020-11-16T21:32:29.398064+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:29.398064+0000","last_undegraded":"2020-11-16T21:32:29.398064+0000","last_fullsized":"2020-11-16T21:32:29.398064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"40","state":"active+clean","last_fresh":"2020-11-16T21:32:21.511649+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:21.511649+0000","last_peered":"2020-11-16T21:32:21.511649+0000","last_clean":"2020-11-16T21:32:21.511649+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:21.511649+0000","last_undegraded":"2020-11-16T21:32:21.511649+0000","last_fullsized":"2020-11-16T21:32:21.511649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12718","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:29.399144+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:29.399144+0000","last_peered":"2020-11-16T21:32:29.399144+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:29.399144+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6704","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.501169+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:20.501169+0000","last_peered":"2020-11-16T21:32:20.501169+0000","last_clean":"2020-11-16T21:32:20.501169+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:20.501169+0000","last_undegraded":"2020-11-16T21:32:20.501169+0000","last_fullsized":"2020-11-16T21:32:20.501169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"15207","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:26.115555+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:26.115555+0000","last_peered":"2020-11-16T21:32:26.115555+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:26.115555+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4149,"num_bytes_recovered":16994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16300","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398187+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:29.398187+0000","last_peered":"2020-11-16T21:32:29.398187+0000","last_clean":"2020-11-16T21:32:29.398187+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:29.398187+0000","last_undegraded":"2020-11-16T21:32:29.398187+0000","last_fullsized":"2020-11-16T21:32:29.398187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28931,"num_bytes_recovered":118497304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497201152,"data_stored":497193008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306780,"kb_used_data":217940,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065060,"statfs":{"total":96636764160,"available":96322621440,"internally_reserved":0,"allocated":223170560,"data_stored":222919273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.31},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.36599999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269672,"kb_used_data":201120,"kb_used_omap":0,"kb_used_meta":68543,"kb_avail":94102168,"statfs":{"total":96636764160,"available":96360620032,"internally_reserved":0,"allocated":205946880,"data_stored":205701351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":70188092},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]}]}} 2020-11-16T21:32:33.562 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:33.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:33.890 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:33.901 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2020-11-16T21:32:32.073623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28931,"num_bytes_recovered":118497304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":576452,"kb_used_data":419060,"kb_used_omap":0,"kb_used_meta":157375,"kb_avail":188167228,"statfs":{"total":193273528320,"available":192683241472,"internally_reserved":0,"allocated":429117440,"data_stored":428620624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":161152060},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1703,"num_bytes_recovered":6975488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001210"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"8151","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:31.114133+0000","last_change":"2020-11-16T21:32:28.720493+0000","last_active":"2020-11-16T21:32:31.114133+0000","last_peered":"2020-11-16T21:32:31.114133+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:31.114133+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":2355200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5469}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12632","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.500375+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:20.500375+0000","last_peered":"2020-11-16T21:32:20.500375+0000","last_clean":"2020-11-16T21:32:20.500375+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:20.500375+0000","last_undegraded":"2020-11-16T21:32:20.500375+0000","last_fullsized":"2020-11-16T21:32:20.500375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16522","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398064+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:29.398064+0000","last_peered":"2020-11-16T21:32:29.398064+0000","last_clean":"2020-11-16T21:32:29.398064+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:29.398064+0000","last_undegraded":"2020-11-16T21:32:29.398064+0000","last_fullsized":"2020-11-16T21:32:29.398064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12759","reported_epoch":"40","state":"active+clean","last_fresh":"2020-11-16T21:32:21.511649+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:21.511649+0000","last_peered":"2020-11-16T21:32:21.511649+0000","last_clean":"2020-11-16T21:32:21.511649+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:21.511649+0000","last_undegraded":"2020-11-16T21:32:21.511649+0000","last_fullsized":"2020-11-16T21:32:21.511649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12718","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:29.399144+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:29.399144+0000","last_peered":"2020-11-16T21:32:29.399144+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:29.399144+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6704","reported_epoch":"39","state":"active+clean","last_fresh":"2020-11-16T21:32:20.501169+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:20.501169+0000","last_peered":"2020-11-16T21:32:20.501169+0000","last_clean":"2020-11-16T21:32:20.501169+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:20.501169+0000","last_undegraded":"2020-11-16T21:32:20.501169+0000","last_fullsized":"2020-11-16T21:32:20.501169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"15207","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:26.115555+0000","last_change":"2020-11-16T21:32:20.437386+0000","last_active":"2020-11-16T21:32:26.115555+0000","last_peered":"2020-11-16T21:32:26.115555+0000","last_clean":"2020-11-16T21:31:26.014543+0000","last_became_active":"2020-11-16T21:31:41.583103+0000","last_became_peered":"2020-11-16T21:31:41.583103+0000","last_unstale":"2020-11-16T21:32:26.115555+0000","last_undegraded":"2020-11-16T21:31:40.435745+0000","last_fullsized":"2020-11-16T21:31:40.395361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4149,"num_bytes_recovered":16994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16300","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398187+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:29.398187+0000","last_peered":"2020-11-16T21:32:29.398187+0000","last_clean":"2020-11-16T21:32:29.398187+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:29.398187+0000","last_undegraded":"2020-11-16T21:32:29.398187+0000","last_fullsized":"2020-11-16T21:32:29.398187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28931,"num_bytes_recovered":118497304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497201152,"data_stored":497193008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306780,"kb_used_data":217940,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065060,"statfs":{"total":96636764160,"available":96322621440,"internally_reserved":0,"allocated":223170560,"data_stored":222919273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.31},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.36599999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269672,"kb_used_data":201120,"kb_used_omap":0,"kb_used_meta":68543,"kb_avail":94102168,"statfs":{"total":96636764160,"available":96360620032,"internally_reserved":0,"allocated":205946880,"data_stored":205701351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":70188092},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]}]}} 2020-11-16T21:32:36.903 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:37.259 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:37.261 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:37.272 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2020-11-16T21:32:36.074241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29695,"num_bytes_recovered":121626648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":587196,"kb_used_data":423788,"kb_used_omap":0,"kb_used_meta":163391,"kb_avail":188156484,"statfs":{"total":193273528320,"available":192672239616,"internally_reserved":0,"allocated":433958912,"data_stored":433450559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":167312444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1339,"num_bytes_recovered":5484544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001184"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"8151","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:31.114133+0000","last_change":"2020-11-16T21:32:28.720493+0000","last_active":"2020-11-16T21:32:31.114133+0000","last_peered":"2020-11-16T21:32:31.114133+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:31.114133+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":2355200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5469}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12634","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.395864+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:29.395864+0000","last_peered":"2020-11-16T21:32:29.395864+0000","last_clean":"2020-11-16T21:32:29.395864+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:29.395864+0000","last_undegraded":"2020-11-16T21:32:29.395864+0000","last_fullsized":"2020-11-16T21:32:29.395864+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16522","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398064+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:29.398064+0000","last_peered":"2020-11-16T21:32:29.398064+0000","last_clean":"2020-11-16T21:32:29.398064+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:29.398064+0000","last_undegraded":"2020-11-16T21:32:29.398064+0000","last_fullsized":"2020-11-16T21:32:29.398064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12761","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.395726+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:29.395726+0000","last_peered":"2020-11-16T21:32:29.395726+0000","last_clean":"2020-11-16T21:32:29.395726+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:29.395726+0000","last_undegraded":"2020-11-16T21:32:29.395726+0000","last_fullsized":"2020-11-16T21:32:29.395726+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12718","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:29.399144+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:29.399144+0000","last_peered":"2020-11-16T21:32:29.399144+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:29.399144+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6706","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.395842+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:29.395842+0000","last_peered":"2020-11-16T21:32:29.395842+0000","last_clean":"2020-11-16T21:32:29.395842+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:29.395842+0000","last_undegraded":"2020-11-16T21:32:29.395842+0000","last_fullsized":"2020-11-16T21:32:29.395842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12712","reported_epoch":"42","state":"active+clean+wait","last_fresh":"2020-11-16T21:32:30.412326+0000","last_change":"2020-11-16T21:32:30.412326+0000","last_active":"2020-11-16T21:32:30.412326+0000","last_peered":"2020-11-16T21:32:30.412326+0000","last_clean":"2020-11-16T21:32:30.412326+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:30.412326+0000","last_undegraded":"2020-11-16T21:32:30.412326+0000","last_fullsized":"2020-11-16T21:32:30.412326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16300","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398187+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:29.398187+0000","last_peered":"2020-11-16T21:32:29.398187+0000","last_clean":"2020-11-16T21:32:29.398187+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:29.398187+0000","last_undegraded":"2020-11-16T21:32:29.398187+0000","last_fullsized":"2020-11-16T21:32:29.398187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29695,"num_bytes_recovered":121626648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501985280,"data_stored":501977136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306780,"kb_used_data":217940,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065060,"statfs":{"total":96636764160,"available":96322621440,"internally_reserved":0,"allocated":223170560,"data_stored":222919273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.31},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.36599999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280416,"kb_used_data":205848,"kb_used_omap":0,"kb_used_meta":74559,"kb_avail":94091424,"statfs":{"total":96636764160,"available":96349618176,"internally_reserved":0,"allocated":210788352,"data_stored":210531286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":76348476},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]}]}]}} 2020-11-16T21:32:37.273 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:37.633 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:37.635 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:37.647 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2020-11-16T21:32:37.413147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30901,"num_bytes_recovered":126566424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":587204,"kb_used_data":423796,"kb_used_omap":0,"kb_used_meta":163391,"kb_avail":188156476,"statfs":{"total":193273528320,"available":192672231424,"internally_reserved":0,"allocated":433967104,"data_stored":433452972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":167312444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2545,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2545,"num_bytes_recovered":10424320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.020979"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"10563","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:36.118334+0000","last_change":"2020-11-16T21:32:28.720493+0000","last_active":"2020-11-16T21:32:36.118334+0000","last_peered":"2020-11-16T21:32:36.118334+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:41.588049+0000","last_became_peered":"2020-11-16T21:31:41.588049+0000","last_unstale":"2020-11-16T21:32:36.118334+0000","last_undegraded":"2020-11-16T21:31:40.441188+0000","last_fullsized":"2020-11-16T21:31:40.395744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1781,"num_bytes_recovered":7294976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6675}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12634","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.395864+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:29.395864+0000","last_peered":"2020-11-16T21:32:29.395864+0000","last_clean":"2020-11-16T21:32:29.395864+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:29.395864+0000","last_undegraded":"2020-11-16T21:32:29.395864+0000","last_fullsized":"2020-11-16T21:32:29.395864+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16522","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398064+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:29.398064+0000","last_peered":"2020-11-16T21:32:29.398064+0000","last_clean":"2020-11-16T21:32:29.398064+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:29.398064+0000","last_undegraded":"2020-11-16T21:32:29.398064+0000","last_fullsized":"2020-11-16T21:32:29.398064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12761","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.395726+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:29.395726+0000","last_peered":"2020-11-16T21:32:29.395726+0000","last_clean":"2020-11-16T21:32:29.395726+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:29.395726+0000","last_undegraded":"2020-11-16T21:32:29.395726+0000","last_fullsized":"2020-11-16T21:32:29.395726+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12718","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:29.399144+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:29.399144+0000","last_peered":"2020-11-16T21:32:29.399144+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:29.399144+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6706","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.395842+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:29.395842+0000","last_peered":"2020-11-16T21:32:29.395842+0000","last_clean":"2020-11-16T21:32:29.395842+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:29.395842+0000","last_undegraded":"2020-11-16T21:32:29.395842+0000","last_fullsized":"2020-11-16T21:32:29.395842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12712","reported_epoch":"42","state":"active+clean+wait","last_fresh":"2020-11-16T21:32:30.412326+0000","last_change":"2020-11-16T21:32:30.412326+0000","last_active":"2020-11-16T21:32:30.412326+0000","last_peered":"2020-11-16T21:32:30.412326+0000","last_clean":"2020-11-16T21:32:30.412326+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:30.412326+0000","last_undegraded":"2020-11-16T21:32:30.412326+0000","last_fullsized":"2020-11-16T21:32:30.412326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16300","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398187+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:29.398187+0000","last_peered":"2020-11-16T21:32:29.398187+0000","last_clean":"2020-11-16T21:32:29.398187+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:29.398187+0000","last_undegraded":"2020-11-16T21:32:29.398187+0000","last_fullsized":"2020-11-16T21:32:29.398187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30901,"num_bytes_recovered":126566424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501985280,"data_stored":501977136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306788,"kb_used_data":217948,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065052,"statfs":{"total":96636764160,"available":96322613248,"internally_reserved":0,"allocated":223178752,"data_stored":222921686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.317},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.35399999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280416,"kb_used_data":205848,"kb_used_omap":0,"kb_used_meta":74559,"kb_avail":94091424,"statfs":{"total":96636764160,"available":96349618176,"internally_reserved":0,"allocated":210788352,"data_stored":210531286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":76348476},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]}]}]}} 2020-11-16T21:32:40.648 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:40.971 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:40.973 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:40.984 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2020-11-16T21:32:40.074860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29120,"num_bytes_recovered":119271448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":580960,"kb_used_data":428048,"kb_used_omap":0,"kb_used_meta":152895,"kb_avail":188162720,"statfs":{"total":193273528320,"available":192678625280,"internally_reserved":0,"allocated":438321152,"data_stored":437801237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":156564540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":774144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001237"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6986","reported_epoch":"43","state":"peering","last_fresh":"2020-11-16T21:32:37.417293+0000","last_change":"2020-11-16T21:32:37.417293+0000","last_active":"2020-11-16T21:31:39.390852+0000","last_peered":"2020-11-16T21:31:38.604129+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:27.539104+0000","last_became_peered":"2020-11-16T21:31:27.539104+0000","last_unstale":"2020-11-16T21:32:37.417293+0000","last_undegraded":"2020-11-16T21:32:37.417293+0000","last_fullsized":"2020-11-16T21:32:37.417293+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12635","reported_epoch":"42","state":"active+clean","last_fresh":"2020-11-16T21:32:30.402225+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:30.402225+0000","last_peered":"2020-11-16T21:32:30.402225+0000","last_clean":"2020-11-16T21:32:30.402225+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:30.402225+0000","last_undegraded":"2020-11-16T21:32:30.402225+0000","last_fullsized":"2020-11-16T21:32:30.402225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16522","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398064+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:29.398064+0000","last_peered":"2020-11-16T21:32:29.398064+0000","last_clean":"2020-11-16T21:32:29.398064+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:29.398064+0000","last_undegraded":"2020-11-16T21:32:29.398064+0000","last_fullsized":"2020-11-16T21:32:29.398064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12762","reported_epoch":"42","state":"active+clean","last_fresh":"2020-11-16T21:32:30.402083+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:30.402083+0000","last_peered":"2020-11-16T21:32:30.402083+0000","last_clean":"2020-11-16T21:32:30.402083+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:30.402083+0000","last_undegraded":"2020-11-16T21:32:30.402083+0000","last_fullsized":"2020-11-16T21:32:30.402083+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12718","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:29.399144+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:29.399144+0000","last_peered":"2020-11-16T21:32:29.399144+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:29.399144+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6707","reported_epoch":"42","state":"active+clean","last_fresh":"2020-11-16T21:32:30.402126+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:30.402126+0000","last_peered":"2020-11-16T21:32:30.402126+0000","last_clean":"2020-11-16T21:32:30.402126+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:30.402126+0000","last_undegraded":"2020-11-16T21:32:30.402126+0000","last_fullsized":"2020-11-16T21:32:30.402126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12714","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416758+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:37.416758+0000","last_peered":"2020-11-16T21:32:37.416758+0000","last_clean":"2020-11-16T21:32:37.416758+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:37.416758+0000","last_undegraded":"2020-11-16T21:32:37.416758+0000","last_fullsized":"2020-11-16T21:32:37.416758+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16300","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398187+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:29.398187+0000","last_peered":"2020-11-16T21:32:29.398187+0000","last_clean":"2020-11-16T21:32:29.398187+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:29.398187+0000","last_undegraded":"2020-11-16T21:32:29.398187+0000","last_fullsized":"2020-11-16T21:32:29.398187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29120,"num_bytes_recovered":119271448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":507461632,"data_stored":507453488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306788,"kb_used_data":217948,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065052,"statfs":{"total":96636764160,"available":96322613248,"internally_reserved":0,"allocated":223178752,"data_stored":222921686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.317},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.35399999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":274172,"kb_used_data":210100,"kb_used_omap":0,"kb_used_meta":64063,"kb_avail":94097668,"statfs":{"total":96636764160,"available":96356012032,"internally_reserved":0,"allocated":215142400,"data_stored":214879551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":65600572},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]}]}]}} 2020-11-16T21:32:40.985 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:41.330 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:41.332 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:41.343 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2020-11-16T21:32:40.074860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29120,"num_bytes_recovered":119271448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":580960,"kb_used_data":428048,"kb_used_omap":0,"kb_used_meta":152895,"kb_avail":188162720,"statfs":{"total":193273528320,"available":192678625280,"internally_reserved":0,"allocated":438321152,"data_stored":437801237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":156564540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":774144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001237"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6986","reported_epoch":"43","state":"peering","last_fresh":"2020-11-16T21:32:37.417293+0000","last_change":"2020-11-16T21:32:37.417293+0000","last_active":"2020-11-16T21:31:39.390852+0000","last_peered":"2020-11-16T21:31:38.604129+0000","last_clean":"2020-11-16T21:31:26.033549+0000","last_became_active":"2020-11-16T21:31:27.539104+0000","last_became_peered":"2020-11-16T21:31:27.539104+0000","last_unstale":"2020-11-16T21:32:37.417293+0000","last_undegraded":"2020-11-16T21:32:37.417293+0000","last_fullsized":"2020-11-16T21:32:37.417293+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12635","reported_epoch":"42","state":"active+clean","last_fresh":"2020-11-16T21:32:30.402225+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:30.402225+0000","last_peered":"2020-11-16T21:32:30.402225+0000","last_clean":"2020-11-16T21:32:30.402225+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:30.402225+0000","last_undegraded":"2020-11-16T21:32:30.402225+0000","last_fullsized":"2020-11-16T21:32:30.402225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16522","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398064+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:29.398064+0000","last_peered":"2020-11-16T21:32:29.398064+0000","last_clean":"2020-11-16T21:32:29.398064+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:29.398064+0000","last_undegraded":"2020-11-16T21:32:29.398064+0000","last_fullsized":"2020-11-16T21:32:29.398064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12762","reported_epoch":"42","state":"active+clean","last_fresh":"2020-11-16T21:32:30.402083+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:30.402083+0000","last_peered":"2020-11-16T21:32:30.402083+0000","last_clean":"2020-11-16T21:32:30.402083+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:30.402083+0000","last_undegraded":"2020-11-16T21:32:30.402083+0000","last_fullsized":"2020-11-16T21:32:30.402083+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12718","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:29.399144+0000","last_change":"2020-11-16T21:31:41.605743+0000","last_active":"2020-11-16T21:32:29.399144+0000","last_peered":"2020-11-16T21:32:29.399144+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:29.399144+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2914,"num_bytes_recovered":11935744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6707","reported_epoch":"42","state":"active+clean","last_fresh":"2020-11-16T21:32:30.402126+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:30.402126+0000","last_peered":"2020-11-16T21:32:30.402126+0000","last_clean":"2020-11-16T21:32:30.402126+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:30.402126+0000","last_undegraded":"2020-11-16T21:32:30.402126+0000","last_fullsized":"2020-11-16T21:32:30.402126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12714","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416758+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:37.416758+0000","last_peered":"2020-11-16T21:32:37.416758+0000","last_clean":"2020-11-16T21:32:37.416758+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:37.416758+0000","last_undegraded":"2020-11-16T21:32:37.416758+0000","last_fullsized":"2020-11-16T21:32:37.416758+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16300","reported_epoch":"41","state":"active+clean","last_fresh":"2020-11-16T21:32:29.398187+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:29.398187+0000","last_peered":"2020-11-16T21:32:29.398187+0000","last_clean":"2020-11-16T21:32:29.398187+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:29.398187+0000","last_undegraded":"2020-11-16T21:32:29.398187+0000","last_fullsized":"2020-11-16T21:32:29.398187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29120,"num_bytes_recovered":119271448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":507461632,"data_stored":507453488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306788,"kb_used_data":217948,"kb_used_omap":0,"kb_used_meta":88832,"kb_avail":94065052,"statfs":{"total":96636764160,"available":96322613248,"internally_reserved":0,"allocated":223178752,"data_stored":222921686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90963968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.317},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.35399999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":274172,"kb_used_data":210100,"kb_used_omap":0,"kb_used_meta":64063,"kb_avail":94097668,"statfs":{"total":96636764160,"available":96356012032,"internally_reserved":0,"allocated":215142400,"data_stored":214879551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":65600572},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]}]}]}} 2020-11-16T21:32:44.345 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:44.686 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:44.688 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:44.700 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2020-11-16T21:32:44.075509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32041,"num_bytes_recovered":131235864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":591532,"kb_used_data":432540,"kb_used_omap":0,"kb_used_meta":158975,"kb_avail":188152148,"statfs":{"total":193273528320,"available":192667799552,"internally_reserved":0,"allocated":442920960,"data_stored":442383654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":162790460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":9609216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001268"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6994","reported_epoch":"44","state":"active+clean","last_fresh":"2020-11-16T21:32:38.429373+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:38.429373+0000","last_peered":"2020-11-16T21:32:38.429373+0000","last_clean":"2020-11-16T21:32:38.429373+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:38.429373+0000","last_undegraded":"2020-11-16T21:32:38.429373+0000","last_fullsized":"2020-11-16T21:32:38.429373+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12636","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416865+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:37.416865+0000","last_peered":"2020-11-16T21:32:37.416865+0000","last_clean":"2020-11-16T21:32:37.416865+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:37.416865+0000","last_undegraded":"2020-11-16T21:32:37.416865+0000","last_fullsized":"2020-11-16T21:32:37.416865+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16524","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.418375+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:37.418375+0000","last_peered":"2020-11-16T21:32:37.418375+0000","last_clean":"2020-11-16T21:32:37.418375+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:37.418375+0000","last_undegraded":"2020-11-16T21:32:37.418375+0000","last_fullsized":"2020-11-16T21:32:37.418375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12763","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416740+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:37.416740+0000","last_peered":"2020-11-16T21:32:37.416740+0000","last_clean":"2020-11-16T21:32:37.416740+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:37.416740+0000","last_undegraded":"2020-11-16T21:32:37.416740+0000","last_fullsized":"2020-11-16T21:32:37.416740+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"14441","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:41.117925+0000","last_change":"2020-11-16T21:32:36.982494+0000","last_active":"2020-11-16T21:32:41.117925+0000","last_peered":"2020-11-16T21:32:41.117925+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:41.117925+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3774,"num_bytes_recovered":15458304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5662}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6708","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416771+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:37.416771+0000","last_peered":"2020-11-16T21:32:37.416771+0000","last_clean":"2020-11-16T21:32:37.416771+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:37.416771+0000","last_undegraded":"2020-11-16T21:32:37.416771+0000","last_fullsized":"2020-11-16T21:32:37.416771+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12715","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.585794+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:37.585794+0000","last_peered":"2020-11-16T21:32:37.585794+0000","last_clean":"2020-11-16T21:32:37.585794+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:37.585794+0000","last_undegraded":"2020-11-16T21:32:37.585794+0000","last_fullsized":"2020-11-16T21:32:37.585794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16302","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.418795+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:37.418795+0000","last_peered":"2020-11-16T21:32:37.418795+0000","last_clean":"2020-11-16T21:32:37.418795+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:37.418795+0000","last_undegraded":"2020-11-16T21:32:37.418795+0000","last_fullsized":"2020-11-16T21:32:37.418795+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32041,"num_bytes_recovered":131235864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":511860736,"data_stored":511852592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306868,"kb_used_data":217964,"kb_used_omap":0,"kb_used_meta":88896,"kb_avail":94064972,"statfs":{"total":96636764160,"available":96322531328,"internally_reserved":0,"allocated":223195136,"data_stored":222926483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91029504},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34300000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284664,"kb_used_data":214576,"kb_used_omap":0,"kb_used_meta":70079,"kb_avail":94087176,"statfs":{"total":96636764160,"available":96345268224,"internally_reserved":0,"allocated":219725824,"data_stored":219457171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":71760956},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.56499999999999995}]}]}]}} 2020-11-16T21:32:44.701 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:45.041 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:45.043 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:45.060 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2020-11-16T21:32:44.075509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32041,"num_bytes_recovered":131235864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":591532,"kb_used_data":432540,"kb_used_omap":0,"kb_used_meta":158975,"kb_avail":188152148,"statfs":{"total":193273528320,"available":192667799552,"internally_reserved":0,"allocated":442920960,"data_stored":442383654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":162790460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":9609216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001268"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6994","reported_epoch":"44","state":"active+clean","last_fresh":"2020-11-16T21:32:38.429373+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:38.429373+0000","last_peered":"2020-11-16T21:32:38.429373+0000","last_clean":"2020-11-16T21:32:38.429373+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:38.429373+0000","last_undegraded":"2020-11-16T21:32:38.429373+0000","last_fullsized":"2020-11-16T21:32:38.429373+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12636","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416865+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:37.416865+0000","last_peered":"2020-11-16T21:32:37.416865+0000","last_clean":"2020-11-16T21:32:37.416865+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:37.416865+0000","last_undegraded":"2020-11-16T21:32:37.416865+0000","last_fullsized":"2020-11-16T21:32:37.416865+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16524","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.418375+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:37.418375+0000","last_peered":"2020-11-16T21:32:37.418375+0000","last_clean":"2020-11-16T21:32:37.418375+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:37.418375+0000","last_undegraded":"2020-11-16T21:32:37.418375+0000","last_fullsized":"2020-11-16T21:32:37.418375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12763","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416740+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:37.416740+0000","last_peered":"2020-11-16T21:32:37.416740+0000","last_clean":"2020-11-16T21:32:37.416740+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:37.416740+0000","last_undegraded":"2020-11-16T21:32:37.416740+0000","last_fullsized":"2020-11-16T21:32:37.416740+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"14441","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2020-11-16T21:32:41.117925+0000","last_change":"2020-11-16T21:32:36.982494+0000","last_active":"2020-11-16T21:32:41.117925+0000","last_peered":"2020-11-16T21:32:41.117925+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:41.117925+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3774,"num_bytes_recovered":15458304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5662}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6708","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416771+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:37.416771+0000","last_peered":"2020-11-16T21:32:37.416771+0000","last_clean":"2020-11-16T21:32:37.416771+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:37.416771+0000","last_undegraded":"2020-11-16T21:32:37.416771+0000","last_fullsized":"2020-11-16T21:32:37.416771+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12715","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.585794+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:37.585794+0000","last_peered":"2020-11-16T21:32:37.585794+0000","last_clean":"2020-11-16T21:32:37.585794+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:37.585794+0000","last_undegraded":"2020-11-16T21:32:37.585794+0000","last_fullsized":"2020-11-16T21:32:37.585794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16302","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.418795+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:37.418795+0000","last_peered":"2020-11-16T21:32:37.418795+0000","last_clean":"2020-11-16T21:32:37.418795+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:37.418795+0000","last_undegraded":"2020-11-16T21:32:37.418795+0000","last_fullsized":"2020-11-16T21:32:37.418795+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32041,"num_bytes_recovered":131235864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":511860736,"data_stored":511852592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306868,"kb_used_data":217964,"kb_used_omap":0,"kb_used_meta":88896,"kb_avail":94064972,"statfs":{"total":96636764160,"available":96322531328,"internally_reserved":0,"allocated":223195136,"data_stored":222926483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91029504},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:30:33 2020","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7039999999999997,"5min":4.7039999999999997,"15min":4.7039999999999997},"last":0.34300000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284664,"kb_used_data":214576,"kb_used_omap":0,"kb_used_meta":70079,"kb_avail":94087176,"statfs":{"total":96636764160,"available":96345268224,"internally_reserved":0,"allocated":219725824,"data_stored":219457171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":71760956},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.56499999999999995}]}]}]}} 2020-11-16T21:32:48.062 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:48.359 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:48.369 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2020-11-16T21:32:48.076341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":591532,"kb_used_data":432540,"kb_used_omap":0,"kb_used_meta":158975,"kb_avail":188152148,"statfs":{"total":193273528320,"available":192667799552,"internally_reserved":0,"allocated":442920960,"data_stored":442383654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":162790460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4088,"num_bytes_recovered":16744448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001481"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6994","reported_epoch":"44","state":"active+clean","last_fresh":"2020-11-16T21:32:38.429373+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:38.429373+0000","last_peered":"2020-11-16T21:32:38.429373+0000","last_clean":"2020-11-16T21:32:38.429373+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:38.429373+0000","last_undegraded":"2020-11-16T21:32:38.429373+0000","last_fullsized":"2020-11-16T21:32:38.429373+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12636","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416865+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:37.416865+0000","last_peered":"2020-11-16T21:32:37.416865+0000","last_clean":"2020-11-16T21:32:37.416865+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:37.416865+0000","last_undegraded":"2020-11-16T21:32:37.416865+0000","last_fullsized":"2020-11-16T21:32:37.416865+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16524","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.418375+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:37.418375+0000","last_peered":"2020-11-16T21:32:37.418375+0000","last_clean":"2020-11-16T21:32:37.418375+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:37.418375+0000","last_undegraded":"2020-11-16T21:32:37.418375+0000","last_fullsized":"2020-11-16T21:32:37.418375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12763","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416740+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:37.416740+0000","last_peered":"2020-11-16T21:32:37.416740+0000","last_clean":"2020-11-16T21:32:37.416740+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:37.416740+0000","last_undegraded":"2020-11-16T21:32:37.416740+0000","last_fullsized":"2020-11-16T21:32:37.416740+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"16776","reported_epoch":"44","state":"active+undersized+remapped","last_fresh":"2020-11-16T21:32:45.513378+0000","last_change":"2020-11-16T21:32:45.513607+0000","last_active":"2020-11-16T21:32:45.513378+0000","last_peered":"2020-11-16T21:32:45.513378+0000","last_clean":"2020-11-16T21:31:26.018939+0000","last_became_active":"2020-11-16T21:31:41.591100+0000","last_became_peered":"2020-11-16T21:31:41.591100+0000","last_unstale":"2020-11-16T21:32:45.513378+0000","last_undegraded":"2020-11-16T21:31:40.439190+0000","last_fullsized":"2020-11-16T21:31:40.395398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"2","objects":6829}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6708","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.416771+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:37.416771+0000","last_peered":"2020-11-16T21:32:37.416771+0000","last_clean":"2020-11-16T21:32:37.416771+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:37.416771+0000","last_undegraded":"2020-11-16T21:32:37.416771+0000","last_fullsized":"2020-11-16T21:32:37.416771+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12715","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.585794+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:37.585794+0000","last_peered":"2020-11-16T21:32:37.585794+0000","last_clean":"2020-11-16T21:32:37.585794+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:37.585794+0000","last_undegraded":"2020-11-16T21:32:37.585794+0000","last_fullsized":"2020-11-16T21:32:37.585794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16302","reported_epoch":"43","state":"active+clean","last_fresh":"2020-11-16T21:32:37.418795+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:37.418795+0000","last_peered":"2020-11-16T21:32:37.418795+0000","last_clean":"2020-11-16T21:32:37.418795+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:37.418795+0000","last_undegraded":"2020-11-16T21:32:37.418795+0000","last_fullsized":"2020-11-16T21:32:37.418795+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":511860736,"data_stored":511852592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306868,"kb_used_data":217964,"kb_used_omap":0,"kb_used_meta":88896,"kb_avail":94064972,"statfs":{"total":96636764160,"available":96322531328,"internally_reserved":0,"allocated":223195136,"data_stored":222926483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91029504},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:32:41 2020","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.34599999999999997}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284664,"kb_used_data":214576,"kb_used_omap":0,"kb_used_meta":70079,"kb_avail":94087176,"statfs":{"total":96636764160,"available":96345268224,"internally_reserved":0,"allocated":219725824,"data_stored":219457171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":71760956},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.56499999999999995}]}]}]}} 2020-11-16T21:32:48.371 INFO:tasks.osd_backfill.ceph_manager:recovered! 2020-11-16T21:32:48.371 INFO:tasks.ceph.osd.0:Restarting daemon 2020-11-16T21:32:48.371 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-11-16T21:32:48.373 INFO:tasks.ceph.osd.0:Started 2020-11-16T21:32:48.374 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-11-16T21:32:48.497 INFO:teuthology.orchestra.run.smithi120.stderr:no valid command found; 10 closest matches: 2020-11-16T21:32:48.497 INFO:teuthology.orchestra.run.smithi120.stderr:0 2020-11-16T21:32:48.497 INFO:teuthology.orchestra.run.smithi120.stderr:1 2020-11-16T21:32:48.498 INFO:teuthology.orchestra.run.smithi120.stderr:2 2020-11-16T21:32:48.498 INFO:teuthology.orchestra.run.smithi120.stderr:abort 2020-11-16T21:32:48.498 INFO:teuthology.orchestra.run.smithi120.stderr:assert 2020-11-16T21:32:48.498 INFO:teuthology.orchestra.run.smithi120.stderr:config diff 2020-11-16T21:32:48.498 INFO:teuthology.orchestra.run.smithi120.stderr:config diff get 2020-11-16T21:32:48.498 INFO:teuthology.orchestra.run.smithi120.stderr:config get 2020-11-16T21:32:48.498 INFO:teuthology.orchestra.run.smithi120.stderr:config help [] 2020-11-16T21:32:48.499 INFO:teuthology.orchestra.run.smithi120.stderr:config set ... 2020-11-16T21:32:48.499 INFO:teuthology.orchestra.run.smithi120.stderr:admin_socket: invalid command 2020-11-16T21:32:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-11-16T21:32:48.500 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2020-11-16T21:32:48.964 INFO:tasks.ceph.osd.0.smithi120.stderr:2020-11-16T21:32:48.972+0000 7f1a61d2bec0 -1 Falling back to public interface 2020-11-16T21:32:52.047 INFO:tasks.ceph.osd.0.smithi120.stderr:2020-11-16T21:32:52.056+0000 7f1a61d2bec0 -1 osd.0 13 log_to_monitors {default=true} 2020-11-16T21:32:53.501 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-11-16T21:32:53.650 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2020-11-16T21:32:53.650 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2020-11-16T21:32:53.650 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2020-11-16T21:32:53.651 INFO:teuthology.orchestra.run.smithi120.stdout:} 2020-11-16T21:32:53.661 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-16T21:32:53.835 INFO:teuthology.orchestra.run.smithi120.stdout:120259084306 2020-11-16T21:32:53.836 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-16T21:32:53.999 INFO:teuthology.orchestra.run.smithi120.stdout:34359738415 2020-11-16T21:32:54.000 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:32:54.336 INFO:teuthology.orchestra.run.smithi120.stdout:120259084304 2020-11-16T21:32:54.348 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084306 got 120259084304 for osd.1 2020-11-16T21:32:55.349 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-16T21:32:55.654 INFO:teuthology.orchestra.run.smithi120.stdout:120259084306 2020-11-16T21:32:55.666 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084306 got 120259084306 for osd.1 2020-11-16T21:32:55.667 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-16T21:32:55.954 INFO:teuthology.orchestra.run.smithi120.stdout:34359738415 2020-11-16T21:32:55.966 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738415 got 34359738415 for osd.2 2020-11-16T21:32:55.967 INFO:tasks.osd_backfill.ceph_manager:waiting for clean 2020-11-16T21:32:55.967 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:56.258 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:56.260 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:56.270 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2020-11-16T21:32:56.077706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600464,"kb_used_data":435968,"kb_used_omap":0,"kb_used_meta":164479,"kb_avail":188143216,"statfs":{"total":193273528320,"available":192658653184,"internally_reserved":0,"allocated":446431232,"data_stored":445865053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":168426556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.587429"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16527","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497586+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:47.497586+0000","last_peered":"2020-11-16T21:32:47.497586+0000","last_clean":"2020-11-16T21:32:47.497586+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:47.497586+0000","last_undegraded":"2020-11-16T21:32:47.497586+0000","last_fullsized":"2020-11-16T21:32:47.497586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12766","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.493964+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:47.493964+0000","last_peered":"2020-11-16T21:32:47.493964+0000","last_clean":"2020-11-16T21:32:47.493964+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:47.493964+0000","last_undegraded":"2020-11-16T21:32:47.493964+0000","last_fullsized":"2020-11-16T21:32:47.493964+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12716","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2020-11-16T21:32:47.506840+0000","last_change":"2020-11-16T21:32:47.505771+0000","last_active":"2020-11-16T21:32:47.506840+0000","last_peered":"2020-11-16T21:32:47.506840+0000","last_clean":"2020-11-16T21:32:47.506840+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:32:47.506840+0000","last_undegraded":"2020-11-16T21:32:47.506840+0000","last_fullsized":"2020-11-16T21:32:47.506840+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6711","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494026+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:47.494026+0000","last_peered":"2020-11-16T21:32:47.494026+0000","last_clean":"2020-11-16T21:32:47.494026+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:47.494026+0000","last_undegraded":"2020-11-16T21:32:47.494026+0000","last_fullsized":"2020-11-16T21:32:47.494026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306884,"kb_used_data":217980,"kb_used_omap":0,"kb_used_meta":88896,"kb_avail":94064956,"statfs":{"total":96636764160,"available":96322514944,"internally_reserved":0,"allocated":223211520,"data_stored":222931230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91029504},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:32:41 2020","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.42999999999999999}]}]},{"osd":0,"up_from":0,"seq":1,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293580,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":75583,"kb_avail":94078260,"statfs":{"total":96636764160,"available":96336138240,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":77397052},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47999999999999998}]}]}]}} 2020-11-16T21:32:56.271 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:56.559 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:56.562 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:56.573 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2020-11-16T21:32:56.077706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600464,"kb_used_data":435968,"kb_used_omap":0,"kb_used_meta":164479,"kb_avail":188143216,"statfs":{"total":193273528320,"available":192658653184,"internally_reserved":0,"allocated":446431232,"data_stored":445865053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":168426556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.587429"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16527","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497586+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:47.497586+0000","last_peered":"2020-11-16T21:32:47.497586+0000","last_clean":"2020-11-16T21:32:47.497586+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:47.497586+0000","last_undegraded":"2020-11-16T21:32:47.497586+0000","last_fullsized":"2020-11-16T21:32:47.497586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12766","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.493964+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:47.493964+0000","last_peered":"2020-11-16T21:32:47.493964+0000","last_clean":"2020-11-16T21:32:47.493964+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:47.493964+0000","last_undegraded":"2020-11-16T21:32:47.493964+0000","last_fullsized":"2020-11-16T21:32:47.493964+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12716","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2020-11-16T21:32:47.506840+0000","last_change":"2020-11-16T21:32:47.505771+0000","last_active":"2020-11-16T21:32:47.506840+0000","last_peered":"2020-11-16T21:32:47.506840+0000","last_clean":"2020-11-16T21:32:47.506840+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:32:47.506840+0000","last_undegraded":"2020-11-16T21:32:47.506840+0000","last_fullsized":"2020-11-16T21:32:47.506840+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6711","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494026+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:47.494026+0000","last_peered":"2020-11-16T21:32:47.494026+0000","last_clean":"2020-11-16T21:32:47.494026+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:47.494026+0000","last_undegraded":"2020-11-16T21:32:47.494026+0000","last_fullsized":"2020-11-16T21:32:47.494026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306884,"kb_used_data":217980,"kb_used_omap":0,"kb_used_meta":88896,"kb_avail":94064956,"statfs":{"total":96636764160,"available":96322514944,"internally_reserved":0,"allocated":223211520,"data_stored":222931230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91029504},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:32:41 2020","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.42999999999999999}]}]},{"osd":0,"up_from":0,"seq":1,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293580,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":75583,"kb_avail":94078260,"statfs":{"total":96636764160,"available":96336138240,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":77397052},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47999999999999998}]}]}]}} 2020-11-16T21:32:56.574 INFO:tasks.osd_backfill.ceph_manager:clean! 2020-11-16T21:32:56.574 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2020-11-16T21:32:56.584 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2020-11-16T21:32:56.584 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:32:56.875 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:56.876 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":47,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:32:52.550586+0000","last_up_change":"2020-11-16T21:32:52.550586+0000","last_in_change":"2020-11-16T21:30:03.448934+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":13,"up_from":47,"up_thru":10,"down_at":14,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":29585}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":29585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":29585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":29585}]},"public_addr":"172.21.15.120:6808/29585","cluster_addr":"172.21.15.120:6809/29585","heartbeat_back_addr":"172.21.15.120:6811/29585","heartbeat_front_addr":"172.21.15.120:6810/29585","state":["exists","up"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":45,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":27113}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":27113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":27113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":27113}]},"public_addr":"172.21.15.120:6800/27113","cluster_addr":"172.21.15.120:6801/27113","heartbeat_back_addr":"172.21.15.120:6803/27113","heartbeat_front_addr":"172.21.15.120:6802/27113","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":65536,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":14},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:32:56.888 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:32:57.172 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:57.174 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:32:57.184 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2020-11-16T21:32:56.077706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600464,"kb_used_data":435968,"kb_used_omap":0,"kb_used_meta":164479,"kb_avail":188143216,"statfs":{"total":193273528320,"available":192658653184,"internally_reserved":0,"allocated":446431232,"data_stored":445865053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":168426556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.587429"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16527","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497586+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:47.497586+0000","last_peered":"2020-11-16T21:32:47.497586+0000","last_clean":"2020-11-16T21:32:47.497586+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:47.497586+0000","last_undegraded":"2020-11-16T21:32:47.497586+0000","last_fullsized":"2020-11-16T21:32:47.497586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12766","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.493964+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:47.493964+0000","last_peered":"2020-11-16T21:32:47.493964+0000","last_clean":"2020-11-16T21:32:47.493964+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:47.493964+0000","last_undegraded":"2020-11-16T21:32:47.493964+0000","last_fullsized":"2020-11-16T21:32:47.493964+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12716","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2020-11-16T21:32:47.506840+0000","last_change":"2020-11-16T21:32:47.505771+0000","last_active":"2020-11-16T21:32:47.506840+0000","last_peered":"2020-11-16T21:32:47.506840+0000","last_clean":"2020-11-16T21:32:47.506840+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:32:47.506840+0000","last_undegraded":"2020-11-16T21:32:47.506840+0000","last_fullsized":"2020-11-16T21:32:47.506840+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6711","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494026+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:47.494026+0000","last_peered":"2020-11-16T21:32:47.494026+0000","last_clean":"2020-11-16T21:32:47.494026+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:47.494026+0000","last_undegraded":"2020-11-16T21:32:47.494026+0000","last_fullsized":"2020-11-16T21:32:47.494026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306884,"kb_used_data":217980,"kb_used_omap":0,"kb_used_meta":88896,"kb_avail":94064956,"statfs":{"total":96636764160,"available":96322514944,"internally_reserved":0,"allocated":223211520,"data_stored":222931230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91029504},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:32:41 2020","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.42999999999999999}]}]},{"osd":0,"up_from":0,"seq":1,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293580,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":75583,"kb_avail":94078260,"statfs":{"total":96636764160,"available":96336138240,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":77397052},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47999999999999998}]}]}]}} 2020-11-16T21:32:57.185 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-16T21:32:57.481 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:32:57.481 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":47,"fsid":"29829f84-fc87-4495-9b6d-a546578bfa29","created":"2020-11-16T21:29:24.399586+0000","modified":"2020-11-16T21:32:52.550586+0000","last_up_change":"2020-11-16T21:32:52.550586+0000","last_in_change":"2020-11-16T21:30:03.448934+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-11-16T21:29:33.080349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"27509b0a-c110-47b7-8822-cf9defe592ac","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":13,"up_from":47,"up_thru":10,"down_at":14,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6808","nonce":29585}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6809","nonce":29585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6811","nonce":29585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6810","nonce":29585}]},"public_addr":"172.21.15.120:6808/29585","cluster_addr":"172.21.15.120:6809/29585","heartbeat_back_addr":"172.21.15.120:6811/29585","heartbeat_front_addr":"172.21.15.120:6810/29585","state":["exists","up"]},{"osd":1,"uuid":"77627eca-b580-425e-b593-30753d001ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":45,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6800","nonce":27113}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6801","nonce":27113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6803","nonce":27113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6802","nonce":27113}]},"public_addr":"172.21.15.120:6800/27113","cluster_addr":"172.21.15.120:6801/27113","heartbeat_back_addr":"172.21.15.120:6803/27113","heartbeat_front_addr":"172.21.15.120:6802/27113","state":["exists","up"]},{"osd":2,"uuid":"05547d39-7932-479e-9303-fc0320537140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6804","nonce":21986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6805","nonce":21986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6807","nonce":21986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.120:6806","nonce":21986}]},"public_addr":"172.21.15.120:6804/21986","cluster_addr":"172.21.15.120:6805/21986","heartbeat_back_addr":"172.21.15.120:6807/21986","heartbeat_front_addr":"172.21.15.120:6806/21986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":65536,"last_purged_snaps_scrub":"2020-11-16T21:29:28.340105+0000","dead_epoch":14},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.392794+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-16T21:29:28.351529+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-16T21:32:58.491 INFO:tasks.ceph:Scrubbing osd.0 2020-11-16T21:32:58.492 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2020-11-16T21:32:58.648 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2020-11-16T21:32:58.648 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-16T21:32:58.648 INFO:teuthology.orchestra.run.smithi120.stdout:} 2020-11-16T21:32:58.658 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2020-11-16T21:32:58.948 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 0 to deep-scrub 2020-11-16T21:32:58.959 INFO:tasks.ceph:Scrubbing osd.1 2020-11-16T21:32:58.959 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2020-11-16T21:32:59.100 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2020-11-16T21:32:59.101 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-16T21:32:59.101 INFO:teuthology.orchestra.run.smithi120.stdout:} 2020-11-16T21:32:59.111 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2020-11-16T21:32:59.404 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 1 to deep-scrub 2020-11-16T21:32:59.414 INFO:tasks.ceph:Scrubbing osd.2 2020-11-16T21:32:59.414 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2020-11-16T21:32:59.554 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2020-11-16T21:32:59.554 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-16T21:32:59.554 INFO:teuthology.orchestra.run.smithi120.stdout:} 2020-11-16T21:32:59.565 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2020-11-16T21:32:59.856 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 2 to deep-scrub 2020-11-16T21:32:59.866 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:33:00.149 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:33:00.151 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:33:00.161 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2020-11-16T21:33:00.078316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600472,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":164479,"kb_avail":188143208,"statfs":{"total":193273528320,"available":192658644992,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":168426556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001640"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16527","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497586+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:47.497586+0000","last_peered":"2020-11-16T21:32:47.497586+0000","last_clean":"2020-11-16T21:32:47.497586+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:47.497586+0000","last_undegraded":"2020-11-16T21:32:47.497586+0000","last_fullsized":"2020-11-16T21:32:47.497586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6849","reported_seq":"12766","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.493964+0000","last_change":"2020-11-16T21:32:21.511649+0000","last_active":"2020-11-16T21:32:47.493964+0000","last_peered":"2020-11-16T21:32:47.493964+0000","last_clean":"2020-11-16T21:32:47.493964+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:32:47.493964+0000","last_undegraded":"2020-11-16T21:32:47.493964+0000","last_fullsized":"2020-11-16T21:32:47.493964+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12716","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2020-11-16T21:32:47.506840+0000","last_change":"2020-11-16T21:32:47.505771+0000","last_active":"2020-11-16T21:32:47.506840+0000","last_peered":"2020-11-16T21:32:47.506840+0000","last_clean":"2020-11-16T21:32:47.506840+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:32:47.506840+0000","last_undegraded":"2020-11-16T21:32:47.506840+0000","last_fullsized":"2020-11-16T21:32:47.506840+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6711","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494026+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:47.494026+0000","last_peered":"2020-11-16T21:32:47.494026+0000","last_clean":"2020-11-16T21:32:47.494026+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:47.494026+0000","last_undegraded":"2020-11-16T21:32:47.494026+0000","last_fullsized":"2020-11-16T21:32:47.494026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54402,"ondisk_log_size":54402,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306892,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":88896,"kb_avail":94064948,"statfs":{"total":96636764160,"available":96322506752,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91029504},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Nov 16 21:32:41 2020","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.495},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.56100000000000005}]}]},{"osd":0,"up_from":0,"seq":201863462914,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293580,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":75583,"kb_avail":94078260,"statfs":{"total":96636764160,"available":96336138240,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":77397052},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.55300000000000005}]}]}]}} 2020-11-16T21:33:00.164 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.164 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.165 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.165 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.165 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.165 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.166 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.166 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:00.166 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:33:20.168 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:33:20.568 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:33:20.570 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:33:20.582 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2020-11-16T21:33:20.081075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57651,"ondisk_log_size":57651,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":618008,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":182015,"kb_avail":188125672,"statfs":{"total":193273528320,"available":192640688128,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":186383420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001661"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16527","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497586+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:47.497586+0000","last_peered":"2020-11-16T21:32:47.497586+0000","last_clean":"2020-11-16T21:32:47.497586+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:47.497586+0000","last_undegraded":"2020-11-16T21:32:47.497586+0000","last_fullsized":"2020-11-16T21:32:47.497586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'12098","reported_seq":"18016","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:33:18.472270+0000","last_change":"2020-11-16T21:33:01.583102+0000","last_active":"2020-11-16T21:33:18.472270+0000","last_peered":"2020-11-16T21:33:18.472270+0000","last_clean":"2020-11-16T21:33:18.472270+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:18.472270+0000","last_undegraded":"2020-11-16T21:33:18.472270+0000","last_fullsized":"2020-11-16T21:33:18.472270+0000","mapping_epoch":39,"log_start":"12'2000","ondisk_log_start":"12'2000","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"30'6829","reported_seq":"12717","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:32:52.559605+0000","last_change":"2020-11-16T21:33:01.594523+0000","last_active":"2020-11-16T21:32:52.559605+0000","last_peered":"2020-11-16T21:32:52.559605+0000","last_clean":"2020-11-16T21:32:52.559605+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:32:52.559605+0000","last_undegraded":"2020-11-16T21:32:52.559605+0000","last_fullsized":"2020-11-16T21:32:52.559605+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6711","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494026+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:47.494026+0000","last_peered":"2020-11-16T21:32:47.494026+0000","last_clean":"2020-11-16T21:32:47.494026+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:47.494026+0000","last_undegraded":"2020-11-16T21:32:47.494026+0000","last_fullsized":"2020-11-16T21:32:47.494026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57651,"ondisk_log_size":57651,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315276,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":97280,"kb_avail":94056564,"statfs":{"total":96636764160,"available":96313921536,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":99614720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:32:41 2020","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.51400000000000001}]}]},{"osd":0,"up_from":0,"seq":201863462918,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302732,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":84735,"kb_avail":94069108,"statfs":{"total":96636764160,"available":96326766592,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":86768700},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.442},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.56299999999999994}]}]}]}} 2020-11-16T21:33:20.584 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.584 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.584 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.584 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.585 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.585 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.585 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.585 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:20.586 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:33:40.587 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:33:40.978 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:33:40.980 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:33:40.993 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2020-11-16T21:33:40.083983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60825,"ondisk_log_size":60825,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616408,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":180415,"kb_avail":188127272,"statfs":{"total":193273528320,"available":192642326528,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":184745020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6788","reported_seq":"16527","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497586+0000","last_change":"2020-11-16T21:32:14.493184+0000","last_active":"2020-11-16T21:32:47.497586+0000","last_peered":"2020-11-16T21:32:47.497586+0000","last_clean":"2020-11-16T21:32:47.497586+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:32:47.497586+0000","last_undegraded":"2020-11-16T21:32:47.497586+0000","last_fullsized":"2020-11-16T21:32:47.497586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6788,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'10603","reported_seq":"16487","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:33:38.588155+0000","last_change":"2020-11-16T21:33:26.835574+0000","last_active":"2020-11-16T21:33:38.588155+0000","last_peered":"2020-11-16T21:33:38.588155+0000","last_clean":"2020-11-16T21:33:38.588155+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:38.588155+0000","last_undegraded":"2020-11-16T21:33:38.588155+0000","last_fullsized":"2020-11-16T21:33:38.588155+0000","mapping_epoch":45,"log_start":"12'600","ondisk_log_start":"12'600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":10003,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6711","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494026+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:47.494026+0000","last_peered":"2020-11-16T21:32:47.494026+0000","last_clean":"2020-11-16T21:32:47.494026+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:47.494026+0000","last_undegraded":"2020-11-16T21:32:47.494026+0000","last_fullsized":"2020-11-16T21:32:47.494026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60825,"ondisk_log_size":60825,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738424,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320652,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":102656,"kb_avail":94051188,"statfs":{"total":96636764160,"available":96308416512,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":105119744},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Mon Nov 16 21:32:41 2020","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.36399999999999999}]}]},{"osd":0,"up_from":0,"seq":201863462922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295756,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":77759,"kb_avail":94076084,"statfs":{"total":96636764160,"available":96333910016,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":79625276},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":2,"last update":"Mon Nov 16 21:32:38 2020","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.628},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.48099999999999998}]}]}]}} 2020-11-16T21:33:40.994 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:40.994 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:40.994 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:40.995 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:40.995 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:40.995 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:40.995 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:33:40.996 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:34:00.997 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:34:01.365 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:34:01.367 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:34:01.379 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2020-11-16T21:34:00.086815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62854,"ondisk_log_size":62854,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":636440,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":200447,"kb_avail":188107240,"statfs":{"total":193273528320,"available":192621813760,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":205257788},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001666"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'8762","reported_seq":"18502","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:33:56.094515+0000","last_change":"2020-11-16T21:33:50.244447+0000","last_active":"2020-11-16T21:33:56.094515+0000","last_peered":"2020-11-16T21:33:56.094515+0000","last_clean":"2020-11-16T21:33:56.094515+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:33:56.094515+0000","last_undegraded":"2020-11-16T21:33:56.094515+0000","last_fullsized":"2020-11-16T21:33:56.094515+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":8762,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6654","reported_seq":"6711","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494026+0000","last_change":"2020-11-16T21:31:51.423845+0000","last_active":"2020-11-16T21:32:47.494026+0000","last_peered":"2020-11-16T21:32:47.494026+0000","last_clean":"2020-11-16T21:32:47.494026+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:32:47.494026+0000","last_undegraded":"2020-11-16T21:32:47.494026+0000","last_fullsized":"2020-11-16T21:32:47.494026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6654,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62854,"ondisk_log_size":62854,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738428,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318220,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":100224,"kb_avail":94053620,"statfs":{"total":96636764160,"available":96310906880,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":102629376},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:33:54 2020","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Nov 16 21:33:44 2020","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.36199999999999999},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.40999999999999998,"15min":0.374},"min":{"1min":0.34599999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.73899999999999999}]}]},{"osd":0,"up_from":0,"seq":201863462926,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084319,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318220,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":100223,"kb_avail":94053620,"statfs":{"total":96636764160,"available":96310906880,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":102628412},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:33:53 2020","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Nov 16 21:33:40 2020","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.40000000000000002,"15min":0.38400000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73299999999999998}]}]}]}} 2020-11-16T21:34:01.380 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:01.380 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:01.380 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:01.381 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:01.381 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:01.381 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:01.381 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:34:21.382 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:34:21.781 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:34:21.783 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:34:21.795 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2020-11-16T21:34:20.089858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65592,"ondisk_log_size":65592,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":647448,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":211455,"kb_avail":188096232,"statfs":{"total":193273528320,"available":192610541568,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":216529980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001900"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'13576","reported_seq":"23318","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:13.556983+0000","last_change":"2020-11-16T21:34:13.557062+0000","last_active":"2020-11-16T21:34:13.556983+0000","last_peered":"2020-11-16T21:34:13.556983+0000","last_clean":"2020-11-16T21:34:13.556983+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:34:13.556983+0000","last_undegraded":"2020-11-16T21:34:13.556983+0000","last_fullsized":"2020-11-16T21:34:13.556983+0000","mapping_epoch":37,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13576","last_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_deep_scrub":"47'13576","last_deep_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_clean_scrub_stamp":"2020-11-16T21:34:13.556975+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"47'8078","reported_seq":"8136","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:34:18.569674+0000","last_change":"2020-11-16T21:34:14.630494+0000","last_active":"2020-11-16T21:34:18.569674+0000","last_peered":"2020-11-16T21:34:18.569674+0000","last_clean":"2020-11-16T21:34:18.569674+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:34:18.569674+0000","last_undegraded":"2020-11-16T21:34:18.569674+0000","last_fullsized":"2020-11-16T21:34:18.569674+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":8078,"ondisk_log_size":8078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"30'6692","reported_seq":"16305","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.497642+0000","last_change":"2020-11-16T21:32:06.471012+0000","last_active":"2020-11-16T21:32:47.497642+0000","last_peered":"2020-11-16T21:32:47.497642+0000","last_clean":"2020-11-16T21:32:47.497642+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:32:47.497642+0000","last_undegraded":"2020-11-16T21:32:47.497642+0000","last_fullsized":"2020-11-16T21:32:47.497642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6692,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65592,"ondisk_log_size":65592,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332172,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":114176,"kb_avail":94039668,"statfs":{"total":96636764160,"available":96296620032,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":116916224},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:33:54 2020","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.379}]},{"osd":1,"last update":"Mon Nov 16 21:33:44 2020","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.36199999999999999},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.40999999999999998,"15min":0.374},"min":{"1min":0.34599999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.53300000000000003}]}]},{"osd":0,"up_from":0,"seq":201863462930,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315276,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":97279,"kb_avail":94056564,"statfs":{"total":96636764160,"available":96313921536,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":99613756},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:33:53 2020","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Nov 16 21:33:40 2020","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.40000000000000002,"15min":0.38400000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.68799999999999994}]}]}]}} 2020-11-16T21:34:21.797 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:21.797 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:21.797 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:21.798 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:21.798 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:21.798 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:34:41.799 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:34:42.167 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:34:42.169 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:34:42.182 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2020-11-16T21:34:42.093063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68296,"ondisk_log_size":68296,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":643544,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":207551,"kb_avail":188100136,"statfs":{"total":193273528320,"available":192614539264,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":212532284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001807"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'13576","reported_seq":"23318","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:13.556983+0000","last_change":"2020-11-16T21:34:13.557062+0000","last_active":"2020-11-16T21:34:13.556983+0000","last_peered":"2020-11-16T21:34:13.556983+0000","last_clean":"2020-11-16T21:34:13.556983+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:34:13.556983+0000","last_undegraded":"2020-11-16T21:34:13.556983+0000","last_fullsized":"2020-11-16T21:34:13.556983+0000","mapping_epoch":37,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13576","last_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_deep_scrub":"47'13576","last_deep_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_clean_scrub_stamp":"2020-11-16T21:34:13.556975+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"47'13308","reported_seq":"13368","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:37.337174+0000","last_change":"2020-11-16T21:34:37.337232+0000","last_active":"2020-11-16T21:34:37.337174+0000","last_peered":"2020-11-16T21:34:37.337174+0000","last_clean":"2020-11-16T21:34:37.337174+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:34:37.337174+0000","last_undegraded":"2020-11-16T21:34:37.337174+0000","last_fullsized":"2020-11-16T21:34:37.337174+0000","mapping_epoch":31,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13308","last_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_deep_scrub":"47'13308","last_deep_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_clean_scrub_stamp":"2020-11-16T21:34:37.337169+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"47'7466","reported_seq":"17080","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:34:41.237236+0000","last_change":"2020-11-16T21:34:39.112024+0000","last_active":"2020-11-16T21:34:41.237236+0000","last_peered":"2020-11-16T21:34:41.237236+0000","last_clean":"2020-11-16T21:34:41.237236+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:34:41.237236+0000","last_undegraded":"2020-11-16T21:34:41.237236+0000","last_fullsized":"2020-11-16T21:34:41.237236+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":7466,"ondisk_log_size":7466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68296,"ondisk_log_size":68296,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738437,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330828,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":112832,"kb_avail":94041012,"statfs":{"total":96636764160,"available":96297996288,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":115539968},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:33:54 2020","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Nov 16 21:33:44 2020","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.36199999999999999},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.40999999999999998,"15min":0.374},"min":{"1min":0.34599999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.60499999999999998}]}]},{"osd":0,"up_from":0,"seq":201863462935,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084327,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312716,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":94719,"kb_avail":94059124,"statfs":{"total":96636764160,"available":96316542976,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":96992316},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:33:53 2020","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Nov 16 21:33:40 2020","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.40000000000000002,"15min":0.38400000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.72399999999999998}]}]}]}} 2020-11-16T21:34:42.183 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:42.183 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:42.184 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:42.184 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:34:42.184 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:35:02.185 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:35:02.521 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:35:02.523 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:35:02.533 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2020-11-16T21:35:02.095790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70846,"ondisk_log_size":70846,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":675096,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":239103,"kb_avail":188068584,"statfs":{"total":193273528320,"available":192582230016,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":244841532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001636"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'13576","reported_seq":"23318","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:13.556983+0000","last_change":"2020-11-16T21:34:13.557062+0000","last_active":"2020-11-16T21:34:13.556983+0000","last_peered":"2020-11-16T21:34:13.556983+0000","last_clean":"2020-11-16T21:34:13.556983+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:34:13.556983+0000","last_undegraded":"2020-11-16T21:34:13.556983+0000","last_fullsized":"2020-11-16T21:34:13.556983+0000","mapping_epoch":37,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13576","last_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_deep_scrub":"47'13576","last_deep_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_clean_scrub_stamp":"2020-11-16T21:34:13.556975+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"47'13308","reported_seq":"13368","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:37.337174+0000","last_change":"2020-11-16T21:34:37.337232+0000","last_active":"2020-11-16T21:34:37.337174+0000","last_peered":"2020-11-16T21:34:37.337174+0000","last_clean":"2020-11-16T21:34:37.337174+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:34:37.337174+0000","last_undegraded":"2020-11-16T21:34:37.337174+0000","last_fullsized":"2020-11-16T21:34:37.337174+0000","mapping_epoch":31,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13308","last_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_deep_scrub":"47'13308","last_deep_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_clean_scrub_stamp":"2020-11-16T21:34:37.337169+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"30'6846","reported_seq":"12718","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494184+0000","last_change":"2020-11-16T21:32:37.585794+0000","last_active":"2020-11-16T21:32:47.494184+0000","last_peered":"2020-11-16T21:32:47.494184+0000","last_clean":"2020-11-16T21:32:47.494184+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:32:47.494184+0000","last_undegraded":"2020-11-16T21:32:47.494184+0000","last_fullsized":"2020-11-16T21:32:47.494184+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6846,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"47'13116","reported_seq":"22730","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:35:01.380676+0000","last_change":"2020-11-16T21:34:39.112024+0000","last_active":"2020-11-16T21:35:01.380676+0000","last_peered":"2020-11-16T21:35:01.380676+0000","last_clean":"2020-11-16T21:35:01.380676+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:35:01.380676+0000","last_undegraded":"2020-11-16T21:35:01.380676+0000","last_fullsized":"2020-11-16T21:35:01.380676+0000","mapping_epoch":35,"log_start":"25'3100","ondisk_log_start":"25'3100","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":10016,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70846,"ondisk_log_size":70846,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738441,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344652,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":126656,"kb_avail":94027188,"statfs":{"total":96636764160,"available":96283840512,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":129695744},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:34:56 2020","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.374,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48299999999999998,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.442}]},{"osd":1,"last update":"Mon Nov 16 21:34:46 2020","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.42299999999999999,"15min":0.374},"min":{"1min":0.39600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71399999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.46000000000000002,"15min":0.39000000000000001},"min":{"1min":0.46200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.876,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.48199999999999998}]}]},{"osd":0,"up_from":0,"seq":201863462939,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084331,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330444,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":112447,"kb_avail":94041396,"statfs":{"total":96636764160,"available":96298389504,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":115145788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:34:56 2020","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.441,"15min":0.44},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Nov 16 21:34:42 2020","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.69699999999999995,"15min":0.48299999999999998},"min":{"1min":0.33800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":25.643999999999998,"5min":25.643999999999998,"15min":25.643999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.032,"5min":0.80700000000000005,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":25.824000000000002,"5min":25.824000000000002,"15min":25.824000000000002},"last":0.59199999999999997}]}]}]}} 2020-11-16T21:35:02.535 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:02.535 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:02.535 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:02.536 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:02.536 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:35:22.537 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:35:22.946 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:35:22.948 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:35:22.959 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2020-11-16T21:35:22.098530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74112,"ondisk_log_size":74112,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":674456,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":238463,"kb_avail":188069224,"statfs":{"total":193273528320,"available":192582885376,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":244186172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001737"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6789","reported_seq":"12639","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494122+0000","last_change":"2020-11-16T21:31:58.448731+0000","last_active":"2020-11-16T21:32:47.494122+0000","last_peered":"2020-11-16T21:32:47.494122+0000","last_clean":"2020-11-16T21:32:47.494122+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:32:47.494122+0000","last_undegraded":"2020-11-16T21:32:47.494122+0000","last_fullsized":"2020-11-16T21:32:47.494122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6789,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'13576","reported_seq":"23318","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:13.556983+0000","last_change":"2020-11-16T21:34:13.557062+0000","last_active":"2020-11-16T21:34:13.556983+0000","last_peered":"2020-11-16T21:34:13.556983+0000","last_clean":"2020-11-16T21:34:13.556983+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:34:13.556983+0000","last_undegraded":"2020-11-16T21:34:13.556983+0000","last_fullsized":"2020-11-16T21:34:13.556983+0000","mapping_epoch":37,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13576","last_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_deep_scrub":"47'13576","last_deep_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_clean_scrub_stamp":"2020-11-16T21:34:13.556975+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"47'13308","reported_seq":"13368","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:37.337174+0000","last_change":"2020-11-16T21:34:37.337232+0000","last_active":"2020-11-16T21:34:37.337174+0000","last_peered":"2020-11-16T21:34:37.337174+0000","last_clean":"2020-11-16T21:34:37.337174+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:34:37.337174+0000","last_undegraded":"2020-11-16T21:34:37.337174+0000","last_fullsized":"2020-11-16T21:34:37.337174+0000","mapping_epoch":31,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13308","last_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_deep_scrub":"47'13308","last_deep_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_clean_scrub_stamp":"2020-11-16T21:34:37.337169+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"47'11545","reported_seq":"17409","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:35:18.797422+0000","last_change":"2020-11-16T21:35:03.796917+0000","last_active":"2020-11-16T21:35:18.797422+0000","last_peered":"2020-11-16T21:35:18.797422+0000","last_clean":"2020-11-16T21:35:18.797422+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:35:18.797422+0000","last_undegraded":"2020-11-16T21:35:18.797422+0000","last_fullsized":"2020-11-16T21:35:18.797422+0000","mapping_epoch":41,"log_start":"12'1500","ondisk_log_start":"12'1500","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":10045,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"47'13383","reported_seq":"22999","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:02.398740+0000","last_change":"2020-11-16T21:35:02.398801+0000","last_active":"2020-11-16T21:35:02.398740+0000","last_peered":"2020-11-16T21:35:02.398740+0000","last_clean":"2020-11-16T21:35:02.398740+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:35:02.398740+0000","last_undegraded":"2020-11-16T21:35:02.398740+0000","last_fullsized":"2020-11-16T21:35:02.398740+0000","mapping_epoch":35,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13383","last_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_deep_scrub":"47'13383","last_deep_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_clean_scrub_stamp":"2020-11-16T21:35:02.398735+0000","log_size":10083,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74112,"ondisk_log_size":74112,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738445,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342156,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":124160,"kb_avail":94029684,"statfs":{"total":96636764160,"available":96286396416,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":127139840},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:34:56 2020","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.374,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48299999999999998,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Nov 16 21:34:46 2020","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.42299999999999999,"15min":0.374},"min":{"1min":0.39600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71399999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.46000000000000002,"15min":0.39000000000000001},"min":{"1min":0.46200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.876,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.61599999999999999}]}]},{"osd":0,"up_from":0,"seq":201863462943,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084335,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332300,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":114303,"kb_avail":94039540,"statfs":{"total":96636764160,"available":96296488960,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":117046332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:34:56 2020","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.441,"15min":0.44},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Nov 16 21:34:42 2020","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.69699999999999995,"15min":0.48299999999999998},"min":{"1min":0.33800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":25.643999999999998,"5min":25.643999999999998,"15min":25.643999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.032,"5min":0.80700000000000005,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":25.824000000000002,"5min":25.824000000000002,"15min":25.824000000000002},"last":0.60799999999999998}]}]}]}} 2020-11-16T21:35:22.960 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:22.960 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:22.961 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:22.961 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:35:42.961 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:35:50.991 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:35:50.992 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:35:51.003 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2020-11-16T21:35:50.102516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77408,"ondisk_log_size":77408,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":690136,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":254143,"kb_avail":188053544,"statfs":{"total":193273528320,"available":192566829056,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":260242492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,10,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001843"},"pg_stats":[{"pgid":"1.7","version":"27'6955","reported_seq":"6997","reported_epoch":"46","state":"active+clean","last_fresh":"2020-11-16T21:32:47.494074+0000","last_change":"2020-11-16T21:32:38.429373+0000","last_active":"2020-11-16T21:32:47.494074+0000","last_peered":"2020-11-16T21:32:47.494074+0000","last_clean":"2020-11-16T21:32:47.494074+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:32:47.494074+0000","last_undegraded":"2020-11-16T21:32:47.494074+0000","last_fullsized":"2020-11-16T21:32:47.494074+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":6955,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'13138","reported_seq":"18965","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:35:48.899846+0000","last_change":"2020-11-16T21:35:27.704021+0000","last_active":"2020-11-16T21:35:48.899846+0000","last_peered":"2020-11-16T21:35:48.899846+0000","last_clean":"2020-11-16T21:35:48.899846+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:35:48.899846+0000","last_undegraded":"2020-11-16T21:35:48.899846+0000","last_fullsized":"2020-11-16T21:35:48.899846+0000","mapping_epoch":33,"log_start":"25'3100","ondisk_log_start":"25'3100","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":10038,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'13576","reported_seq":"23318","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:13.556983+0000","last_change":"2020-11-16T21:34:13.557062+0000","last_active":"2020-11-16T21:34:13.556983+0000","last_peered":"2020-11-16T21:34:13.556983+0000","last_clean":"2020-11-16T21:34:13.556983+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:34:13.556983+0000","last_undegraded":"2020-11-16T21:34:13.556983+0000","last_fullsized":"2020-11-16T21:34:13.556983+0000","mapping_epoch":37,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13576","last_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_deep_scrub":"47'13576","last_deep_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_clean_scrub_stamp":"2020-11-16T21:34:13.556975+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"47'13308","reported_seq":"13368","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:37.337174+0000","last_change":"2020-11-16T21:34:37.337232+0000","last_active":"2020-11-16T21:34:37.337174+0000","last_peered":"2020-11-16T21:34:37.337174+0000","last_clean":"2020-11-16T21:34:37.337174+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:34:37.337174+0000","last_undegraded":"2020-11-16T21:34:37.337174+0000","last_fullsized":"2020-11-16T21:34:37.337174+0000","mapping_epoch":31,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13308","last_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_deep_scrub":"47'13308","last_deep_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_clean_scrub_stamp":"2020-11-16T21:34:37.337169+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"47'13692","reported_seq":"19567","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:26.820161+0000","last_change":"2020-11-16T21:35:26.820228+0000","last_active":"2020-11-16T21:35:26.820161+0000","last_peered":"2020-11-16T21:35:26.820161+0000","last_clean":"2020-11-16T21:35:26.820161+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:35:26.820161+0000","last_undegraded":"2020-11-16T21:35:26.820161+0000","last_fullsized":"2020-11-16T21:35:26.820161+0000","mapping_epoch":41,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13692","last_scrub_stamp":"2020-11-16T21:35:26.820155+0000","last_deep_scrub":"47'13692","last_deep_scrub_stamp":"2020-11-16T21:35:26.820155+0000","last_clean_scrub_stamp":"2020-11-16T21:35:26.820155+0000","log_size":10092,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"47'13383","reported_seq":"22999","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:02.398740+0000","last_change":"2020-11-16T21:35:02.398801+0000","last_active":"2020-11-16T21:35:02.398740+0000","last_peered":"2020-11-16T21:35:02.398740+0000","last_clean":"2020-11-16T21:35:02.398740+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:35:02.398740+0000","last_undegraded":"2020-11-16T21:35:02.398740+0000","last_fullsized":"2020-11-16T21:35:02.398740+0000","mapping_epoch":35,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13383","last_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_deep_scrub":"47'13383","last_deep_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_clean_scrub_stamp":"2020-11-16T21:35:02.398735+0000","log_size":10083,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77408,"ondisk_log_size":77408,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738450,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345228,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":127232,"kb_avail":94026612,"statfs":{"total":96636764160,"available":96283250688,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":130285568},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:34:56 2020","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.374,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48299999999999998,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.432}]},{"osd":1,"last update":"Mon Nov 16 21:34:46 2020","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.42299999999999999,"15min":0.374},"min":{"1min":0.39600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71399999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.46000000000000002,"15min":0.39000000000000001},"min":{"1min":0.46200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.876,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.67600000000000005}]}]},{"osd":0,"up_from":0,"seq":201863462948,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084341,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344908,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":126911,"kb_avail":94026932,"statfs":{"total":96636764160,"available":96283578368,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":129956924},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:34:56 2020","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.441,"15min":0.44},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.499},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Nov 16 21:35:46 2020","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.72699999999999998,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.755,"5min":25.643999999999998,"15min":25.643999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.84999999999999998,"15min":0.59199999999999997},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1559999999999999,"5min":25.824000000000002,"15min":25.824000000000002},"last":0.64400000000000002}]}]}]}} 2020-11-16T21:35:51.004 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:51.005 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:35:51.005 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:36:11.006 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:36:11.496 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:36:11.497 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:36:11.509 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2020-11-16T21:36:10.105356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80572,"ondisk_log_size":80572,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":715992,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":279999,"kb_avail":188027688,"statfs":{"total":193273528320,"available":192540352512,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":286719036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"1.7","version":"47'11279","reported_seq":"11299","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-16T21:36:09.116653+0000","last_change":"2020-11-16T21:35:54.758762+0000","last_active":"2020-11-16T21:36:09.116653+0000","last_peered":"2020-11-16T21:36:09.116653+0000","last_clean":"2020-11-16T21:36:09.116653+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:36:09.116653+0000","last_undegraded":"2020-11-16T21:36:09.116653+0000","last_fullsized":"2020-11-16T21:36:09.116653+0000","mapping_epoch":43,"log_start":"12'1200","ondisk_log_start":"12'1200","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-16T21:29:33.579234+0000","last_clean_scrub_stamp":"2020-11-16T21:29:33.579234+0000","log_size":10079,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'13578","reported_seq":"19431","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:50.662944+0000","last_change":"2020-11-16T21:35:50.663008+0000","last_active":"2020-11-16T21:35:50.662944+0000","last_peered":"2020-11-16T21:35:50.662944+0000","last_clean":"2020-11-16T21:35:50.662944+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:35:50.662944+0000","last_undegraded":"2020-11-16T21:35:50.662944+0000","last_fullsized":"2020-11-16T21:35:50.662944+0000","mapping_epoch":33,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13578","last_scrub_stamp":"2020-11-16T21:35:50.662940+0000","last_deep_scrub":"47'13578","last_deep_scrub_stamp":"2020-11-16T21:35:50.662940+0000","last_clean_scrub_stamp":"2020-11-16T21:35:50.662940+0000","log_size":10078,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'13576","reported_seq":"23318","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:13.556983+0000","last_change":"2020-11-16T21:34:13.557062+0000","last_active":"2020-11-16T21:34:13.556983+0000","last_peered":"2020-11-16T21:34:13.556983+0000","last_clean":"2020-11-16T21:34:13.556983+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:34:13.556983+0000","last_undegraded":"2020-11-16T21:34:13.556983+0000","last_fullsized":"2020-11-16T21:34:13.556983+0000","mapping_epoch":37,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13576","last_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_deep_scrub":"47'13576","last_deep_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_clean_scrub_stamp":"2020-11-16T21:34:13.556975+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"47'13308","reported_seq":"13368","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:37.337174+0000","last_change":"2020-11-16T21:34:37.337232+0000","last_active":"2020-11-16T21:34:37.337174+0000","last_peered":"2020-11-16T21:34:37.337174+0000","last_clean":"2020-11-16T21:34:37.337174+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:34:37.337174+0000","last_undegraded":"2020-11-16T21:34:37.337174+0000","last_fullsized":"2020-11-16T21:34:37.337174+0000","mapping_epoch":31,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13308","last_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_deep_scrub":"47'13308","last_deep_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_clean_scrub_stamp":"2020-11-16T21:34:37.337169+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"47'13692","reported_seq":"19567","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:26.820161+0000","last_change":"2020-11-16T21:35:26.820228+0000","last_active":"2020-11-16T21:35:26.820161+0000","last_peered":"2020-11-16T21:35:26.820161+0000","last_clean":"2020-11-16T21:35:26.820161+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:35:26.820161+0000","last_undegraded":"2020-11-16T21:35:26.820161+0000","last_fullsized":"2020-11-16T21:35:26.820161+0000","mapping_epoch":41,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13692","last_scrub_stamp":"2020-11-16T21:35:26.820155+0000","last_deep_scrub":"47'13692","last_deep_scrub_stamp":"2020-11-16T21:35:26.820155+0000","last_clean_scrub_stamp":"2020-11-16T21:35:26.820155+0000","log_size":10092,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"47'13383","reported_seq":"22999","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:02.398740+0000","last_change":"2020-11-16T21:35:02.398801+0000","last_active":"2020-11-16T21:35:02.398740+0000","last_peered":"2020-11-16T21:35:02.398740+0000","last_clean":"2020-11-16T21:35:02.398740+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:35:02.398740+0000","last_undegraded":"2020-11-16T21:35:02.398740+0000","last_fullsized":"2020-11-16T21:35:02.398740+0000","mapping_epoch":35,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13383","last_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_deep_scrub":"47'13383","last_deep_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_clean_scrub_stamp":"2020-11-16T21:35:02.398735+0000","log_size":10083,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80572,"ondisk_log_size":80572,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738454,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362252,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":144256,"kb_avail":94009588,"statfs":{"total":96636764160,"available":96265818112,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":147718144},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:36:00 2020","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.497,"15min":0.497},"min":{"1min":0.374,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80900000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Nov 16 21:35:47 2020","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.47599999999999998,"15min":0.39200000000000002},"min":{"1min":0.40899999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.51500000000000001,"15min":0.40899999999999997},"min":{"1min":0.45000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.93300000000000005,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.65800000000000003}]}]},{"osd":0,"up_from":0,"seq":201863462952,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084345,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353740,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":135743,"kb_avail":94018100,"statfs":{"total":96636764160,"available":96274534400,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":139000892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:35:58 2020","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.45600000000000002,"15min":0.44500000000000001},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51200000000000001,"15min":0.51400000000000001},"min":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73999999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.435}]},{"osd":2,"last update":"Mon Nov 16 21:35:46 2020","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.72699999999999998,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.755,"5min":25.643999999999998,"15min":25.643999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.84999999999999998,"15min":0.59199999999999997},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1559999999999999,"5min":25.824000000000002,"15min":25.824000000000002},"last":0.79600000000000004}]}]}]}} 2020-11-16T21:36:11.511 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-16T21:29:33.579234+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=29, tm_sec=33, tm_wday=0, tm_yday=321, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=16, tm_hour=21, tm_min=32, tm_sec=57, tm_wday=0, tm_yday=321, tm_isdst=0) 2020-11-16T21:36:11.511 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-16T21:36:31.513 INFO:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-16T21:36:31.811 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-11-16T21:36:31.813 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2020-11-16T21:36:31.824 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2020-11-16T21:36:30.108898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80503,"ondisk_log_size":80503,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":709976,"kb_used_data":435976,"kb_used_omap":0,"kb_used_meta":273983,"kb_avail":188033704,"statfs":{"total":193273528320,"available":192546512896,"internally_reserved":0,"allocated":446439424,"data_stored":445867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":280558652},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002436"},"pg_stats":[{"pgid":"1.7","version":"47'13910","reported_seq":"13955","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:36:19.052887+0000","last_change":"2020-11-16T21:36:19.052971+0000","last_active":"2020-11-16T21:36:19.052887+0000","last_peered":"2020-11-16T21:36:19.052887+0000","last_clean":"2020-11-16T21:36:19.052887+0000","last_became_active":"2020-11-16T21:32:38.424220+0000","last_became_peered":"2020-11-16T21:32:38.424220+0000","last_unstale":"2020-11-16T21:36:19.052887+0000","last_undegraded":"2020-11-16T21:36:19.052887+0000","last_fullsized":"2020-11-16T21:36:19.052887+0000","mapping_epoch":43,"log_start":"25'3900","ondisk_log_start":"25'3900","created":10,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13910","last_scrub_stamp":"2020-11-16T21:36:19.052879+0000","last_deep_scrub":"47'13910","last_deep_scrub_stamp":"2020-11-16T21:36:19.052879+0000","last_clean_scrub_stamp":"2020-11-16T21:36:19.052879+0000","log_size":10010,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28487680,"num_objects":6955,"num_object_clones":0,"num_object_copies":13910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":8441856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'13578","reported_seq":"19431","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:50.662944+0000","last_change":"2020-11-16T21:35:50.663008+0000","last_active":"2020-11-16T21:35:50.662944+0000","last_peered":"2020-11-16T21:35:50.662944+0000","last_clean":"2020-11-16T21:35:50.662944+0000","last_became_active":"2020-11-16T21:31:58.443788+0000","last_became_peered":"2020-11-16T21:31:58.443788+0000","last_unstale":"2020-11-16T21:35:50.662944+0000","last_undegraded":"2020-11-16T21:35:50.662944+0000","last_fullsized":"2020-11-16T21:35:50.662944+0000","mapping_epoch":33,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13578","last_scrub_stamp":"2020-11-16T21:35:50.662940+0000","last_deep_scrub":"47'13578","last_deep_scrub_stamp":"2020-11-16T21:35:50.662940+0000","last_clean_scrub_stamp":"2020-11-16T21:35:50.662940+0000","log_size":10078,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27807744,"num_objects":6789,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":19927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"47'13576","reported_seq":"23318","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:13.556983+0000","last_change":"2020-11-16T21:34:13.557062+0000","last_active":"2020-11-16T21:34:13.556983+0000","last_peered":"2020-11-16T21:34:13.556983+0000","last_clean":"2020-11-16T21:34:13.556983+0000","last_became_active":"2020-11-16T21:32:14.486550+0000","last_became_peered":"2020-11-16T21:32:14.486550+0000","last_unstale":"2020-11-16T21:34:13.556983+0000","last_undegraded":"2020-11-16T21:34:13.556983+0000","last_fullsized":"2020-11-16T21:34:13.556983+0000","mapping_epoch":37,"log_start":"25'3500","ondisk_log_start":"25'3500","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13576","last_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_deep_scrub":"47'13576","last_deep_scrub_stamp":"2020-11-16T21:34:13.556975+0000","last_clean_scrub_stamp":"2020-11-16T21:34:13.556975+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4828,"num_bytes_recovered":19775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'13698","reported_seq":"19618","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:24.437509+0000","last_change":"2020-11-16T21:33:24.437585+0000","last_active":"2020-11-16T21:33:24.437509+0000","last_peered":"2020-11-16T21:33:24.437509+0000","last_clean":"2020-11-16T21:33:24.437509+0000","last_became_active":"2020-11-16T21:32:21.506859+0000","last_became_peered":"2020-11-16T21:32:21.506859+0000","last_unstale":"2020-11-16T21:33:24.437509+0000","last_undegraded":"2020-11-16T21:33:24.437509+0000","last_fullsized":"2020-11-16T21:33:24.437509+0000","mapping_epoch":39,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13698","last_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_deep_scrub":"47'13698","last_deep_scrub_stamp":"2020-11-16T21:33:24.437504+0000","last_clean_scrub_stamp":"2020-11-16T21:33:24.437504+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28053504,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4905,"num_bytes_recovered":20090880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'13658","reported_seq":"19549","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:33:49.781678+0000","last_change":"2020-11-16T21:33:49.781756+0000","last_active":"2020-11-16T21:33:49.781678+0000","last_peered":"2020-11-16T21:33:49.781678+0000","last_clean":"2020-11-16T21:33:49.781678+0000","last_became_active":"2020-11-16T21:32:47.500080+0000","last_became_peered":"2020-11-16T21:32:47.500080+0000","last_unstale":"2020-11-16T21:33:49.781678+0000","last_undegraded":"2020-11-16T21:33:49.781678+0000","last_fullsized":"2020-11-16T21:33:49.781678+0000","mapping_epoch":45,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13658","last_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_deep_scrub":"47'13658","last_deep_scrub_stamp":"2020-11-16T21:33:49.781670+0000","last_clean_scrub_stamp":"2020-11-16T21:33:49.781670+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27971584,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4941,"num_bytes_recovered":20238336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"47'13308","reported_seq":"13368","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:34:37.337174+0000","last_change":"2020-11-16T21:34:37.337232+0000","last_active":"2020-11-16T21:34:37.337174+0000","last_peered":"2020-11-16T21:34:37.337174+0000","last_clean":"2020-11-16T21:34:37.337174+0000","last_became_active":"2020-11-16T21:31:51.419323+0000","last_became_peered":"2020-11-16T21:31:51.419323+0000","last_unstale":"2020-11-16T21:34:37.337174+0000","last_undegraded":"2020-11-16T21:34:37.337174+0000","last_fullsized":"2020-11-16T21:34:37.337174+0000","mapping_epoch":31,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13308","last_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_deep_scrub":"47'13308","last_deep_scrub_stamp":"2020-11-16T21:34:37.337169+0000","last_clean_scrub_stamp":"2020-11-16T21:34:37.337169+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"47'13692","reported_seq":"19567","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:26.820161+0000","last_change":"2020-11-16T21:35:26.820228+0000","last_active":"2020-11-16T21:35:26.820161+0000","last_peered":"2020-11-16T21:35:26.820161+0000","last_clean":"2020-11-16T21:35:26.820161+0000","last_became_active":"2020-11-16T21:32:30.407131+0000","last_became_peered":"2020-11-16T21:32:30.407131+0000","last_unstale":"2020-11-16T21:35:26.820161+0000","last_undegraded":"2020-11-16T21:35:26.820161+0000","last_fullsized":"2020-11-16T21:35:26.820161+0000","mapping_epoch":41,"log_start":"25'3600","ondisk_log_start":"25'3600","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13692","last_scrub_stamp":"2020-11-16T21:35:26.820155+0000","last_deep_scrub":"47'13692","last_deep_scrub_stamp":"2020-11-16T21:35:26.820155+0000","last_clean_scrub_stamp":"2020-11-16T21:35:26.820155+0000","log_size":10092,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28041216,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":27384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4913,"num_bytes_recovered":20123648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"47'13383","reported_seq":"22999","reported_epoch":"47","state":"active+clean","last_fresh":"2020-11-16T21:35:02.398740+0000","last_change":"2020-11-16T21:35:02.398801+0000","last_active":"2020-11-16T21:35:02.398740+0000","last_peered":"2020-11-16T21:35:02.398740+0000","last_clean":"2020-11-16T21:35:02.398740+0000","last_became_active":"2020-11-16T21:32:06.464950+0000","last_became_peered":"2020-11-16T21:32:06.464950+0000","last_unstale":"2020-11-16T21:35:02.398740+0000","last_undegraded":"2020-11-16T21:35:02.398740+0000","last_fullsized":"2020-11-16T21:35:02.398740+0000","mapping_epoch":35,"log_start":"25'3300","ondisk_log_start":"25'3300","created":10,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13383","last_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_deep_scrub":"47'13383","last_deep_scrub_stamp":"2020-11-16T21:35:02.398735+0000","last_clean_scrub_stamp":"2020-11-16T21:35:02.398735+0000","log_size":10083,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27402264,"num_objects":6691,"num_object_clones":0,"num_object_copies":13382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":26762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":19513368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":222822424,"num_objects":54401,"num_object_clones":0,"num_object_copies":108802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54402,"num_write_kb":217602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33208,"num_bytes_recovered":136015896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516427776,"data_stored":516415560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80503,"ondisk_log_size":80503,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365708,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":147712,"kb_avail":94006132,"statfs":{"total":96636764160,"available":96262279168,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":151257088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:36:00 2020","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.497,"15min":0.497},"min":{"1min":0.374,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80900000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Nov 16 21:35:47 2020","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.47599999999999998,"15min":0.39200000000000002},"min":{"1min":0.40899999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.51500000000000001,"15min":0.40899999999999997},"min":{"1min":0.45000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.93300000000000005,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.63}]}]},{"osd":0,"up_from":0,"seq":201863462956,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084349,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344268,"kb_used_data":217988,"kb_used_omap":0,"kb_used_meta":126271,"kb_avail":94027572,"statfs":{"total":96636764160,"available":96284233728,"internally_reserved":0,"allocated":223219712,"data_stored":222933823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964,"internal_metadata":129301564},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Nov 16 21:35:58 2020","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.45600000000000002,"15min":0.44500000000000001},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51200000000000001,"15min":0.51400000000000001},"min":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73999999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Nov 16 21:35:46 2020","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.72699999999999998,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.755,"5min":25.643999999999998,"15min":25.643999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.84999999999999998,"15min":0.59199999999999997},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1559999999999999,"5min":25.824000000000002,"15min":25.824000000000002},"last":0.71999999999999997}]}]}]}} 2020-11-16T21:36:31.825 INFO:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2020-11-16T21:36:32.133 INFO:teuthology.misc:Shutting down mds daemons... 2020-11-16T21:36:32.134 INFO:teuthology.misc:Shutting down osd daemons... 2020-11-16T21:36:32.134 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2020-11-16T21:36:32.135 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:36:32.169 INFO:tasks.ceph.osd.0:Stopped 2020-11-16T21:36:32.170 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2020-11-16T21:36:32.170 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:36:32.251 INFO:tasks.ceph.osd.1:Stopped 2020-11-16T21:36:32.252 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2020-11-16T21:36:32.252 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:36:32.307 INFO:tasks.ceph.osd.2:Stopped 2020-11-16T21:36:32.308 INFO:teuthology.misc:Shutting down mgr daemons... 2020-11-16T21:36:32.308 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2020-11-16T21:36:32.309 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:36:32.324 INFO:tasks.ceph.mgr.x:Stopped 2020-11-16T21:36:32.324 INFO:teuthology.misc:Shutting down mon daemons... 2020-11-16T21:36:32.325 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2020-11-16T21:36:32.325 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:36:32.374 INFO:tasks.ceph.mon.a:Stopped 2020-11-16T21:36:32.375 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-16T21:36:32.375 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:36:32.424 INFO:tasks.ceph.mon.b:Stopped 2020-11-16T21:36:32.425 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-16T21:36:32.425 INFO:teuthology.orchestra.run:waiting for 300 2020-11-16T21:36:32.474 INFO:tasks.ceph.mon.c:Stopped 2020-11-16T21:36:32.475 INFO:tasks.ceph:Checking cluster log for badness... 2020-11-16T21:36:32.475 INFO:teuthology.orchestra.run.smithi120:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'Health detail' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2020-11-16T21:36:32.554 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:36:32.555 INFO:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2020-11-16T21:36:38.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:36:38.406 INFO:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-16T21:36:38.538 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi120.front.sepia.ceph.com 2020-11-16T21:36:38.538 INFO:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2020-11-16T21:36:38.598 INFO:tasks.ceph:Archiving mon data... 2020-11-16T21:36:38.600 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154/data/mon.a.tgz 2020-11-16T21:36:38.600 INFO:teuthology.orchestra.run.smithi120:> mktemp 2020-11-16T21:36:38.631 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.kMWIgyXY1O 2020-11-16T21:36:38.632 INFO:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kMWIgyXY1O 2020-11-16T21:36:38.901 INFO:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.kMWIgyXY1O 2020-11-16T21:36:39.029 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.kMWIgyXY1O is 623KB 2020-11-16T21:36:39.097 INFO:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.kMWIgyXY1O 2020-11-16T21:36:39.110 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154/data/mon.b.tgz 2020-11-16T21:36:39.111 INFO:teuthology.orchestra.run.smithi120:> mktemp 2020-11-16T21:36:39.163 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.XQeT0gfF4X 2020-11-16T21:36:39.163 INFO:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.XQeT0gfF4X 2020-11-16T21:36:39.449 INFO:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.XQeT0gfF4X 2020-11-16T21:36:39.533 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.XQeT0gfF4X is 985KB 2020-11-16T21:36:39.634 INFO:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.XQeT0gfF4X 2020-11-16T21:36:39.648 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154/data/mon.c.tgz 2020-11-16T21:36:39.648 INFO:teuthology.orchestra.run.smithi120:> mktemp 2020-11-16T21:36:39.700 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.pNNHaFl7Ub 2020-11-16T21:36:39.701 INFO:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.pNNHaFl7Ub 2020-11-16T21:36:39.983 INFO:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.pNNHaFl7Ub 2020-11-16T21:36:40.066 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.pNNHaFl7Ub is 962KB 2020-11-16T21:36:40.166 INFO:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.pNNHaFl7Ub 2020-11-16T21:36:40.179 INFO:tasks.ceph:Cleaning ceph cluster... 2020-11-16T21:36:40.180 INFO:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-16T21:36:40.245 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2020-11-16T21:36:40.246 INFO:teuthology.orchestra.run.smithi120:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2020-11-16T21:36:40.313 INFO:teuthology.orchestra.run.smithi120.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2020-11-16T21:36:40.319 INFO:tasks.ceph:Archiving crash dumps... 2020-11-16T21:36:40.323 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/crash to /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154/remote/smithi120/crash 2020-11-16T21:36:40.324 INFO:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2020-11-16T21:36:40.387 INFO:tasks.ceph:Compressing logs... 2020-11-16T21:36:40.387 INFO:teuthology.orchestra.run.smithi120:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2020-11-16T21:41:39.583 INFO:tasks.ceph:Archiving logs... 2020-11-16T21:41:39.585 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631154/remote/smithi120/log 2020-11-16T21:41:39.586 INFO:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/log/ceph -- .